2023-01-11T20:41:05.2069703Z Requested labels: linux.2xlarge
2023-01-11T20:41:05.2069803Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/91627/merge
2023-01-11T20:41:05.2069976Z Reusable workflow chain:
2023-01-11T20:41:05.2070023Z pytorch/pytorch/.github/workflows/pull.yml@refs/pull/91627/merge (57fc38f02f250896a12b32cfa200a6105a03d09c)
2023-01-11T20:41:05.2070067Z -> pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/91627/merge (57fc38f02f250896a12b32cfa200a6105a03d09c)
2023-01-11T20:41:05.2070095Z Waiting for a runner to pick up this job...
2023-01-11T20:41:05.7777255Z Job is about to start running on the runner: i-03e06aa264bd8cb8d (organization)
2023-01-11T20:41:11.4093357Z Current runner version: '2.300.2'
2023-01-11T20:41:11.4098023Z Runner name: 'i-03e06aa264bd8cb8d'
2023-01-11T20:41:11.4098379Z Runner group name: 'Default'
2023-01-11T20:41:11.4098832Z Machine name: 'ip-10-0-3-131'
2023-01-11T20:41:11.4100113Z ##[group]GITHUB_TOKEN Permissions
2023-01-11T20:41:11.4100563Z Actions: read
2023-01-11T20:41:11.4100726Z Checks: read
2023-01-11T20:41:11.4100900Z Contents: read
2023-01-11T20:41:11.4101082Z Deployments: read
2023-01-11T20:41:11.4101250Z Discussions: read
2023-01-11T20:41:11.4101422Z Issues: read
2023-01-11T20:41:11.4101601Z Metadata: read
2023-01-11T20:41:11.4101918Z Packages: read
2023-01-11T20:41:11.4102087Z Pages: read
2023-01-11T20:41:11.4102261Z PullRequests: read
2023-01-11T20:41:11.4102455Z RepositoryProjects: read
2023-01-11T20:41:11.4102656Z SecurityEvents: read
2023-01-11T20:41:11.4102839Z Statuses: read
2023-01-11T20:41:11.4102997Z ##[endgroup]
2023-01-11T20:41:11.4106103Z Secret source: None
2023-01-11T20:41:11.4106484Z Prepare workflow directory
2023-01-11T20:41:11.6380748Z Prepare all required actions
2023-01-11T20:41:11.6553379Z Getting action download info
2023-01-11T20:41:11.8536711Z Download action repository 'pytorch/test-infra@main' (SHA:2c225610d00fb13c04fcd60389d3e4d8326167c3)
2023-01-11T20:41:12.1154975Z Download action repository 'pytorch/pytorch@master' (SHA:c5836153f5332ca83d5cacde38f2829a4d54793e)
2023-01-11T20:41:14.7327800Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2023-01-11T20:41:14.9913808Z Getting action download info
2023-01-11T20:41:15.1608508Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2023-01-11T20:41:15.3062968Z Getting action download info
2023-01-11T20:41:15.4614175Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2023-01-11T20:41:15.5951565Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2023-01-11T20:41:15.5953672Z ##[group] Inputs
2023-01-11T20:41:15.5954000Z   build-environment: linux-focal-py3.7-gcc7
2023-01-11T20:41:15.5954821Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 2, runner: "linux.2xlarge" },
  { config: "default", shard: 2, num_shards: 2, runner: "linux.2xlarge" },
  { config: "distributed", shard: 1, num_shards: 2, runner: "linux.2xlarge" },
  { config: "distributed", shard: 2, num_shards: 2, runner: "linux.2xlarge" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "docs_test", shard: 1, num_shards: 1,  runner: "linux.2xlarge" },
  { config: "jit_legacy", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "backwards_compat", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
]}

2023-01-11T20:41:15.5955697Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:41:15.5956084Z   sync-tag: 
2023-01-11T20:41:15.5956838Z   timeout-minutes: 240
2023-01-11T20:41:15.5957063Z   use-gha: 
2023-01-11T20:41:15.5957291Z ##[endgroup]
2023-01-11T20:41:15.5957777Z Complete job name: linux-focal-py3.7-gcc7 / test (distributed, 1, 2, linux.2xlarge)
2023-01-11T20:41:15.6648939Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2023-01-11T20:41:15.6649264Z with:
2023-01-11T20:41:15.6649829Z   github-secret: ***
2023-01-11T20:41:15.6650208Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2023-01-11T20:41:15.6650562Z   activate-with-label: false
2023-01-11T20:41:15.6650791Z   label: with-ssh
2023-01-11T20:41:15.6651013Z   remove-existing-keys: true
2023-01-11T20:41:15.6651218Z env:
2023-01-11T20:41:15.6651431Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:41:15.6651650Z ##[endgroup]
2023-01-11T20:41:16.1129908Z Grabbing public ssh keys from https://github.com/LucaLumetti.keys
2023-01-11T20:41:16.2008178Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2023-01-11T20:41:16.2020743Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2023-01-11T20:41:16.2045044Z Login using: ssh ec2-user@ec2-18-209-66-237.compute-1.amazonaws.com
2023-01-11T20:41:16.2045633Z All testing is done inside the container, to start an interactive session run:
2023-01-11T20:41:16.2046157Z   docker exec -it $(docker container ps --format '{{.ID}}') bash
2023-01-11T20:41:16.2291796Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2023-01-11T20:41:16.2292070Z with:
2023-01-11T20:41:16.2292253Z   submodules: recursive
2023-01-11T20:41:16.2292426Z   fetch-depth: 0
2023-01-11T20:41:16.2292592Z env:
2023-01-11T20:41:16.2292761Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:41:16.2292932Z ##[endgroup]
2023-01-11T20:41:16.2506179Z ##[group]Run retry () {
2023-01-11T20:41:16.2506460Z retry () {
2023-01-11T20:41:16.2506690Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2023-01-11T20:41:16.2506917Z }
2023-01-11T20:41:16.2507092Z echo "${GITHUB_WORKSPACE}"
2023-01-11T20:41:16.2507311Z if [ -z "${NO_SUDO}" ]; then
2023-01-11T20:41:16.2507538Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2023-01-11T20:41:16.2507728Z else
2023-01-11T20:41:16.2507925Z   retry rm -rf "${GITHUB_WORKSPACE}"
2023-01-11T20:41:16.2508117Z fi
2023-01-11T20:41:16.2508323Z mkdir "${GITHUB_WORKSPACE}"
2023-01-11T20:41:16.2524221Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:41:16.2524462Z env:
2023-01-11T20:41:16.2524678Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:41:16.2524863Z   NO_SUDO: 
2023-01-11T20:41:16.2525017Z ##[endgroup]
2023-01-11T20:41:16.2619162Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T20:41:18.6714634Z ##[group]Run malfet/checkout@silent-checkout
2023-01-11T20:41:18.6714921Z with:
2023-01-11T20:41:18.6715156Z   ref: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T20:41:18.6715412Z   fetch-depth: 0
2023-01-11T20:41:18.6715617Z   submodules: recursive
2023-01-11T20:41:18.6715839Z   quiet-checkout: true
2023-01-11T20:41:18.6716071Z   repository: pytorch/pytorch
2023-01-11T20:41:18.6716447Z   token: ***
2023-01-11T20:41:18.6716636Z   ssh-strict: true
2023-01-11T20:41:18.6716861Z   persist-credentials: true
2023-01-11T20:41:18.6717087Z   clean: true
2023-01-11T20:41:18.6717280Z   lfs: false
2023-01-11T20:41:18.6717493Z   set-safe-directory: true
2023-01-11T20:41:18.6717702Z env:
2023-01-11T20:41:18.6717889Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:41:18.6718100Z ##[endgroup]
2023-01-11T20:41:18.7840868Z Syncing repository: pytorch/pytorch
2023-01-11T20:41:18.7842317Z ##[group]Getting Git version info
2023-01-11T20:41:18.7842800Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2023-01-11T20:41:18.7843295Z [command]/usr/bin/git version
2023-01-11T20:41:18.7843510Z git version 2.38.1
2023-01-11T20:41:18.7844237Z ##[endgroup]
2023-01-11T20:41:18.7855619Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/257a4287-2837-4d1c-91d2-608236a42222' before making global git config changes
2023-01-11T20:41:18.7856070Z Adding repository directory to the temporary git global config as a safe directory
2023-01-11T20:41:18.7856576Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T20:41:18.7888063Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2023-01-11T20:41:18.7892430Z ##[group]Initializing the repository
2023-01-11T20:41:18.7895305Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T20:41:18.7924758Z hint: Using 'master' as the name for the initial branch. This default branch name
2023-01-11T20:41:18.7925424Z hint: is subject to change. To configure the initial branch name to use in all
2023-01-11T20:41:18.7925986Z hint: of your new repositories, which will suppress this warning, call:
2023-01-11T20:41:18.7926405Z hint: 
2023-01-11T20:41:18.7926699Z hint: 	git config --global init.defaultBranch <name>
2023-01-11T20:41:18.7926945Z hint: 
2023-01-11T20:41:18.7927264Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2023-01-11T20:41:18.7927653Z hint: 'development'. The just-created branch can be renamed via this command:
2023-01-11T20:41:18.7927920Z hint: 
2023-01-11T20:41:18.7928232Z hint: 	git branch -m <name>
2023-01-11T20:41:18.7928828Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2023-01-11T20:41:18.7934825Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2023-01-11T20:41:18.7963562Z ##[endgroup]
2023-01-11T20:41:18.7963967Z ##[group]Disabling automatic garbage collection
2023-01-11T20:41:18.7966781Z [command]/usr/bin/git config --local gc.auto 0
2023-01-11T20:41:18.7992499Z ##[endgroup]
2023-01-11T20:41:18.7993331Z ##[group]Setting up auth
2023-01-11T20:41:18.7998835Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-01-11T20:41:18.8027628Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2023-01-11T20:41:18.8267997Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-01-11T20:41:18.8298808Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2023-01-11T20:41:18.8543202Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-01-11T20:41:18.8589808Z ##[endgroup]
2023-01-11T20:41:18.8590219Z ##[group]Fetching the repository
2023-01-11T20:41:18.8595834Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2023-01-11T20:42:14.3584745Z [command]/usr/bin/git rev-parse --verify --quiet 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e^{object}
2023-01-11T20:42:14.3611479Z 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T20:42:14.3617100Z ##[endgroup]
2023-01-11T20:42:14.3617761Z ##[group]Determining the checkout info
2023-01-11T20:42:14.3618892Z ##[endgroup]
2023-01-11T20:42:14.3619506Z ##[group]Checking out the ref
2023-01-11T20:42:14.3622931Z [command]/usr/bin/git checkout --quiet --force 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T20:42:15.6577002Z ##[endgroup]
2023-01-11T20:42:15.6577538Z ##[group]Setting up auth for fetching submodules
2023-01-11T20:42:15.6583486Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-01-11T20:42:15.6630224Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2023-01-11T20:42:15.6659717Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2023-01-11T20:42:15.6686447Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2023-01-11T20:42:15.6711542Z ##[endgroup]
2023-01-11T20:42:15.6711995Z ##[group]Fetching submodules
2023-01-11T20:42:15.6715491Z [command]/usr/bin/git submodule sync --recursive
2023-01-11T20:42:15.6979133Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2023-01-11T20:42:15.7237633Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2023-01-11T20:42:15.7238280Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2023-01-11T20:42:15.7240593Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2023-01-11T20:42:15.7243109Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2023-01-11T20:42:15.7246071Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2023-01-11T20:42:15.7248877Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2023-01-11T20:42:15.7251698Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2023-01-11T20:42:15.7254683Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2023-01-11T20:42:15.7257762Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2023-01-11T20:42:15.7261461Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2023-01-11T20:42:15.7265035Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2023-01-11T20:42:15.7268390Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2023-01-11T20:42:15.7272040Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2023-01-11T20:42:15.7275620Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2023-01-11T20:42:15.7279319Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2023-01-11T20:42:15.7283052Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2023-01-11T20:42:15.7287081Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2023-01-11T20:42:15.7291084Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2023-01-11T20:42:15.7295108Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2023-01-11T20:42:15.7299766Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2023-01-11T20:42:15.7304345Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2023-01-11T20:42:15.7308759Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2023-01-11T20:42:15.7313357Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2023-01-11T20:42:15.7317952Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2023-01-11T20:42:15.7322656Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2023-01-11T20:42:15.7327663Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2023-01-11T20:42:15.7332661Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2023-01-11T20:42:15.7337643Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2023-01-11T20:42:15.7343157Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2023-01-11T20:42:15.7348294Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2023-01-11T20:42:15.7353649Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2023-01-11T20:42:15.7359053Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2023-01-11T20:42:15.7364778Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2023-01-11T20:42:15.7370391Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2023-01-11T20:42:15.7376368Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2023-01-11T20:42:15.7383711Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2023-01-11T20:42:15.7389751Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2023-01-11T20:42:15.7396598Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2023-01-11T20:42:15.7402642Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2023-01-11T20:42:15.7409462Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2023-01-11T20:42:15.7415795Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2023-01-11T20:42:15.7443705Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2023-01-11T20:42:16.0795626Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2023-01-11T20:42:16.3097454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2023-01-11T20:42:16.5410793Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2023-01-11T20:42:16.9087621Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2023-01-11T20:42:17.2103543Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2023-01-11T20:42:19.4940757Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2023-01-11T20:42:26.0574134Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2023-01-11T20:42:26.4791483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2023-01-11T20:42:26.9887440Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2023-01-11T20:42:28.4383911Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2023-01-11T20:42:29.5244571Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2023-01-11T20:42:30.7692380Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2023-01-11T20:42:36.6228304Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2023-01-11T20:42:37.3432537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2023-01-11T20:42:39.0266634Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2023-01-11T20:42:40.2402874Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2023-01-11T20:42:40.4590236Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2023-01-11T20:42:40.8905128Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2023-01-11T20:42:41.1960264Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2023-01-11T20:42:42.1910037Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2023-01-11T20:42:42.6714900Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2023-01-11T20:42:42.9065154Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2023-01-11T20:42:43.1788952Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2023-01-11T20:42:44.6372038Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2023-01-11T20:42:45.0107004Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2023-01-11T20:42:45.3809363Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2023-01-11T20:42:51.2413398Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2023-01-11T20:42:53.2858507Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2023-01-11T20:42:53.7659472Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2023-01-11T20:42:54.0046857Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2023-01-11T20:42:59.6460032Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2023-01-11T20:42:59.8739103Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2023-01-11T20:43:00.1115239Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2023-01-11T20:43:00.9998567Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2023-01-11T20:43:01.3446376Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2023-01-11T20:43:01.8009678Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2023-01-11T20:43:02.1367780Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2023-01-11T20:43:02.7201151Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2023-01-11T20:43:06.5966139Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2023-01-11T20:43:07.1223444Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2023-01-11T20:43:09.4372704Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2023-01-11T20:43:09.4469969Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2023-01-11T20:43:09.4545135Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2023-01-11T20:43:09.4750720Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2023-01-11T20:43:09.4955904Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2023-01-11T20:43:09.5301430Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2023-01-11T20:43:10.0882848Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2023-01-11T20:43:10.1076526Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-01-11T20:43:10.1990990Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2023-01-11T20:43:10.2308300Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2023-01-11T20:43:10.4965686Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2023-01-11T20:43:10.8762353Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2023-01-11T20:43:11.0988340Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2023-01-11T20:43:11.1413268Z Submodule path 'third_party/fbgemm': checked out '80d64206c07879fd4683be66873de7cefa1a0a71'
2023-01-11T20:43:11.1427415Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2023-01-11T20:43:11.1429225Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T20:43:11.1431348Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2023-01-11T20:43:11.1433672Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T20:43:11.1456476Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2023-01-11T20:43:12.3633722Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2023-01-11T20:43:13.0173480Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2023-01-11T20:43:14.0233957Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2023-01-11T20:43:14.3890641Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2023-01-11T20:43:14.4819785Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2023-01-11T20:43:14.5358312Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2023-01-11T20:43:14.5455580Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2023-01-11T20:43:14.6232546Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2023-01-11T20:43:14.6575369Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b'
2023-01-11T20:43:14.6662764Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2023-01-11T20:43:14.7035058Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2023-01-11T20:43:14.7254197Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195'
2023-01-11T20:43:14.7679476Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2023-01-11T20:43:14.7793329Z Submodule path 'third_party/ideep': checked out 'e533c771a1e75a1c225c14b2261eefa62681d9e6'
2023-01-11T20:43:14.7806326Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2023-01-11T20:43:14.7829302Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2023-01-11T20:43:23.1829782Z Submodule path 'third_party/ideep/mkl-dnn': checked out '404ad76ee633c939d705eb583ffe50a806969d5e'
2023-01-11T20:43:23.1846344Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T20:43:23.1870948Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2023-01-11T20:43:31.6998699Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out 'fbec3e25a559ee252022ae066817b204e106a6ba'
2023-01-11T20:43:31.7092584Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2023-01-11T20:43:31.7224258Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2023-01-11T20:43:31.8081438Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562'
2023-01-11T20:43:31.8096259Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T20:43:31.8098385Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T20:43:31.8121841Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2023-01-11T20:43:33.0452621Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2023-01-11T20:43:34.2358288Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2023-01-11T20:43:34.2858859Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2023-01-11T20:43:34.3055683Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2023-01-11T20:43:34.3188837Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2023-01-11T20:43:34.4166274Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2023-01-11T20:43:34.6463037Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2023-01-11T20:43:34.6488247Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2023-01-11T20:43:34.6490605Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2023-01-11T20:43:34.6515474Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2023-01-11T20:43:35.0651203Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2023-01-11T20:43:35.9723081Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-01-11T20:43:36.0011208Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2023-01-11T20:43:36.0148438Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2023-01-11T20:43:36.0160268Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T20:43:36.0183190Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2023-01-11T20:43:38.3614502Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2023-01-11T20:43:38.3634373Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T20:43:38.3636208Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T20:43:38.3661567Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2023-01-11T20:43:38.9535387Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2023-01-11T20:43:39.8302458Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2023-01-11T20:43:39.8926089Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2023-01-11T20:43:39.8940330Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T20:43:39.8962655Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2023-01-11T20:43:40.1396991Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-01-11T20:43:40.1479661Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2023-01-11T20:43:40.3854106Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2023-01-11T20:43:40.3873392Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2023-01-11T20:43:40.3875417Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2023-01-11T20:43:40.3898399Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2023-01-11T20:43:40.8573459Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2023-01-11T20:43:41.8974899Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2023-01-11T20:43:41.9610528Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2023-01-11T20:43:41.9689986Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2023-01-11T20:43:41.9787425Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2023-01-11T20:43:42.0090963Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917'
2023-01-11T20:43:42.0171780Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2023-01-11T20:43:42.0433477Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2023-01-11T20:43:42.0519303Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2023-01-11T20:43:42.0922836Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2023-01-11T20:43:42.1937923Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2023-01-11T20:43:42.2172378Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2023-01-11T20:43:42.2186976Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2023-01-11T20:43:42.2188849Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2023-01-11T20:43:42.2190897Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2023-01-11T20:43:42.2193063Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T20:43:42.2216015Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2023-01-11T20:43:43.2393962Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2023-01-11T20:43:43.5173215Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2023-01-11T20:43:44.6845974Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2023-01-11T20:43:45.6107286Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2023-01-11T20:43:45.6238750Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2023-01-11T20:43:45.6816815Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2023-01-11T20:43:45.7070541Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2023-01-11T20:43:45.7084212Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T20:43:45.7111652Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2023-01-11T20:43:45.9601729Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-01-11T20:43:46.0852959Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2023-01-11T20:43:46.0880906Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2023-01-11T20:43:46.1147730Z Entering 'android/libs/fbjni'
2023-01-11T20:43:46.1183122Z Entering 'third_party/FP16'
2023-01-11T20:43:46.1217619Z Entering 'third_party/FXdiv'
2023-01-11T20:43:46.1252186Z Entering 'third_party/NNPACK'
2023-01-11T20:43:46.1286972Z Entering 'third_party/QNNPACK'
2023-01-11T20:43:46.1322514Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T20:43:46.1358946Z Entering 'third_party/XNNPACK'
2023-01-11T20:43:46.1404155Z Entering 'third_party/benchmark'
2023-01-11T20:43:46.1438979Z Entering 'third_party/cpuinfo'
2023-01-11T20:43:46.1474192Z Entering 'third_party/cub'
2023-01-11T20:43:46.1509493Z Entering 'third_party/cudnn_frontend'
2023-01-11T20:43:46.1550448Z Entering 'third_party/cutlass'
2023-01-11T20:43:46.1592495Z Entering 'third_party/eigen'
2023-01-11T20:43:46.1630244Z Entering 'third_party/fbgemm'
2023-01-11T20:43:46.1668001Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T20:43:46.1703235Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T20:43:46.1737045Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T20:43:46.1770889Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T20:43:46.1805723Z Entering 'third_party/flatbuffers'
2023-01-11T20:43:46.1841931Z Entering 'third_party/fmt'
2023-01-11T20:43:46.1876570Z Entering 'third_party/foxi'
2023-01-11T20:43:46.1911917Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T20:43:46.1946309Z Entering 'third_party/gloo'
2023-01-11T20:43:46.1980140Z Entering 'third_party/googletest'
2023-01-11T20:43:46.2014227Z Entering 'third_party/ideep'
2023-01-11T20:43:46.2047882Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T20:43:46.2083634Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T20:43:46.2124873Z Entering 'third_party/ios-cmake'
2023-01-11T20:43:46.2159551Z Entering 'third_party/ittapi'
2023-01-11T20:43:46.2193528Z Entering 'third_party/kineto'
2023-01-11T20:43:46.2230227Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T20:43:46.2263853Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T20:43:46.2299078Z Entering 'third_party/nccl/nccl'
2023-01-11T20:43:46.2333201Z Entering 'third_party/neon2sse'
2023-01-11T20:43:46.2367304Z Entering 'third_party/nlohmann'
2023-01-11T20:43:46.2403335Z Entering 'third_party/onnx'
2023-01-11T20:43:46.2450721Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T20:43:46.2485042Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T20:43:46.2520988Z Entering 'third_party/onnx-tensorrt'
2023-01-11T20:43:46.2555366Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T20:43:46.2593451Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T20:43:46.2628457Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T20:43:46.2661479Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T20:43:46.2699163Z Entering 'third_party/pocketfft'
2023-01-11T20:43:46.2733350Z Entering 'third_party/protobuf'
2023-01-11T20:43:46.2770969Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T20:43:46.2804637Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T20:43:46.2839850Z Entering 'third_party/psimd'
2023-01-11T20:43:46.2874561Z Entering 'third_party/pthreadpool'
2023-01-11T20:43:46.2908568Z Entering 'third_party/pybind11'
2023-01-11T20:43:46.2942810Z Entering 'third_party/python-enum'
2023-01-11T20:43:46.2975721Z Entering 'third_party/python-peachpy'
2023-01-11T20:43:46.3010167Z Entering 'third_party/python-six'
2023-01-11T20:43:46.3044110Z Entering 'third_party/sleef'
2023-01-11T20:43:46.3079637Z Entering 'third_party/tbb'
2023-01-11T20:43:46.3116858Z Entering 'third_party/tensorpipe'
2023-01-11T20:43:46.3150967Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T20:43:46.3185544Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T20:43:46.3218755Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T20:43:46.3252538Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T20:43:46.3286024Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T20:43:46.3321309Z Entering 'third_party/zstd'
2023-01-11T20:43:46.3363725Z ##[endgroup]
2023-01-11T20:43:46.3365055Z ##[group]Persisting credentials for submodules
2023-01-11T20:43:46.3369454Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2023-01-11T20:43:46.3624723Z Entering 'android/libs/fbjni'
2023-01-11T20:43:46.3658603Z Entering 'third_party/FP16'
2023-01-11T20:43:46.3692927Z Entering 'third_party/FXdiv'
2023-01-11T20:43:46.3727014Z Entering 'third_party/NNPACK'
2023-01-11T20:43:46.3761583Z Entering 'third_party/QNNPACK'
2023-01-11T20:43:46.3796022Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T20:43:46.3830190Z Entering 'third_party/XNNPACK'
2023-01-11T20:43:46.3874796Z Entering 'third_party/benchmark'
2023-01-11T20:43:46.3909288Z Entering 'third_party/cpuinfo'
2023-01-11T20:43:46.3943527Z Entering 'third_party/cub'
2023-01-11T20:43:46.3977200Z Entering 'third_party/cudnn_frontend'
2023-01-11T20:43:46.4016524Z Entering 'third_party/cutlass'
2023-01-11T20:43:46.4057108Z Entering 'third_party/eigen'
2023-01-11T20:43:46.4093559Z Entering 'third_party/fbgemm'
2023-01-11T20:43:46.4127516Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T20:43:46.4161693Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T20:43:46.4195626Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T20:43:46.4228818Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T20:43:46.4263418Z Entering 'third_party/flatbuffers'
2023-01-11T20:43:46.4298827Z Entering 'third_party/fmt'
2023-01-11T20:43:46.4333555Z Entering 'third_party/foxi'
2023-01-11T20:43:46.4366577Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T20:43:46.4400627Z Entering 'third_party/gloo'
2023-01-11T20:43:46.4434381Z Entering 'third_party/googletest'
2023-01-11T20:43:46.4469119Z Entering 'third_party/ideep'
2023-01-11T20:43:46.4502926Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T20:43:46.4538253Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T20:43:46.4578358Z Entering 'third_party/ios-cmake'
2023-01-11T20:43:46.4613059Z Entering 'third_party/ittapi'
2023-01-11T20:43:46.4647660Z Entering 'third_party/kineto'
2023-01-11T20:43:46.4682072Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T20:43:46.4715782Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T20:43:46.4751612Z Entering 'third_party/nccl/nccl'
2023-01-11T20:43:46.4786537Z Entering 'third_party/neon2sse'
2023-01-11T20:43:46.4819778Z Entering 'third_party/nlohmann'
2023-01-11T20:43:46.4855082Z Entering 'third_party/onnx'
2023-01-11T20:43:46.4900655Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T20:43:46.4936240Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T20:43:46.4971319Z Entering 'third_party/onnx-tensorrt'
2023-01-11T20:43:46.5004240Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T20:43:46.5044024Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T20:43:46.5077413Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T20:43:46.5110261Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T20:43:46.5147875Z Entering 'third_party/pocketfft'
2023-01-11T20:43:46.5181535Z Entering 'third_party/protobuf'
2023-01-11T20:43:46.5219498Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T20:43:46.5253452Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T20:43:46.5288380Z Entering 'third_party/psimd'
2023-01-11T20:43:46.5322200Z Entering 'third_party/pthreadpool'
2023-01-11T20:43:46.5356755Z Entering 'third_party/pybind11'
2023-01-11T20:43:46.5390634Z Entering 'third_party/python-enum'
2023-01-11T20:43:46.5425133Z Entering 'third_party/python-peachpy'
2023-01-11T20:43:46.5459408Z Entering 'third_party/python-six'
2023-01-11T20:43:46.5494614Z Entering 'third_party/sleef'
2023-01-11T20:43:46.5528464Z Entering 'third_party/tbb'
2023-01-11T20:43:46.5564593Z Entering 'third_party/tensorpipe'
2023-01-11T20:43:46.5599121Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T20:43:46.5633194Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T20:43:46.5665975Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T20:43:46.5699747Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T20:43:46.5732434Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T20:43:46.5768110Z Entering 'third_party/zstd'
2023-01-11T20:43:46.5813624Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2023-01-11T20:43:46.6071334Z Entering 'android/libs/fbjni'
2023-01-11T20:43:46.6102330Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2023-01-11T20:43:46.6115875Z Entering 'third_party/FP16'
2023-01-11T20:43:46.6148663Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2023-01-11T20:43:46.6162588Z Entering 'third_party/FXdiv'
2023-01-11T20:43:46.6194451Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2023-01-11T20:43:46.6208959Z Entering 'third_party/NNPACK'
2023-01-11T20:43:46.6241263Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2023-01-11T20:43:46.6255373Z Entering 'third_party/QNNPACK'
2023-01-11T20:43:46.6286932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2023-01-11T20:43:46.6301246Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T20:43:46.6333489Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2023-01-11T20:43:46.6347829Z Entering 'third_party/XNNPACK'
2023-01-11T20:43:46.6379007Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2023-01-11T20:43:46.6403514Z Entering 'third_party/benchmark'
2023-01-11T20:43:46.6436228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T20:43:46.6450949Z Entering 'third_party/cpuinfo'
2023-01-11T20:43:46.6482494Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2023-01-11T20:43:46.6497361Z Entering 'third_party/cub'
2023-01-11T20:43:46.6529583Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2023-01-11T20:43:46.6543810Z Entering 'third_party/cudnn_frontend'
2023-01-11T20:43:46.6575493Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2023-01-11T20:43:46.6594483Z Entering 'third_party/cutlass'
2023-01-11T20:43:46.6626051Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2023-01-11T20:43:46.6646893Z Entering 'third_party/eigen'
2023-01-11T20:43:46.6678879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2023-01-11T20:43:46.6695752Z Entering 'third_party/fbgemm'
2023-01-11T20:43:46.6729598Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2023-01-11T20:43:46.6745048Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T20:43:46.6776576Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2023-01-11T20:43:46.6791395Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T20:43:46.6824273Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2023-01-11T20:43:46.6838564Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T20:43:46.6870422Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2023-01-11T20:43:46.6884345Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T20:43:46.6916263Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2023-01-11T20:43:46.6931584Z Entering 'third_party/flatbuffers'
2023-01-11T20:43:46.6964906Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2023-01-11T20:43:46.6981386Z Entering 'third_party/fmt'
2023-01-11T20:43:46.7014430Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2023-01-11T20:43:46.7029252Z Entering 'third_party/foxi'
2023-01-11T20:43:46.7060976Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2023-01-11T20:43:46.7075120Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T20:43:46.7109174Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2023-01-11T20:43:46.7123645Z Entering 'third_party/gloo'
2023-01-11T20:43:46.7155782Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2023-01-11T20:43:46.7170122Z Entering 'third_party/googletest'
2023-01-11T20:43:46.7201523Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2023-01-11T20:43:46.7216595Z Entering 'third_party/ideep'
2023-01-11T20:43:46.7251128Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2023-01-11T20:43:46.7265153Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T20:43:46.7296875Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2023-01-11T20:43:46.7314251Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T20:43:46.7346707Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2023-01-11T20:43:46.7366768Z Entering 'third_party/ios-cmake'
2023-01-11T20:43:46.7399789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2023-01-11T20:43:46.7414234Z Entering 'third_party/ittapi'
2023-01-11T20:43:46.7448235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2023-01-11T20:43:46.7461816Z Entering 'third_party/kineto'
2023-01-11T20:43:46.7493162Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2023-01-11T20:43:46.7507611Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T20:43:46.7538982Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2023-01-11T20:43:46.7554370Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T20:43:46.7586167Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2023-01-11T20:43:46.7602195Z Entering 'third_party/nccl/nccl'
2023-01-11T20:43:46.7633918Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2023-01-11T20:43:46.7648702Z Entering 'third_party/neon2sse'
2023-01-11T20:43:46.7681254Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2023-01-11T20:43:46.7695039Z Entering 'third_party/nlohmann'
2023-01-11T20:43:46.7727328Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2023-01-11T20:43:46.7742150Z Entering 'third_party/onnx'
2023-01-11T20:43:46.7773418Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2023-01-11T20:43:46.7798922Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T20:43:46.7831342Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T20:43:46.7846024Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T20:43:46.7878006Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T20:43:46.7894679Z Entering 'third_party/onnx-tensorrt'
2023-01-11T20:43:46.7927468Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2023-01-11T20:43:46.7941295Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T20:43:46.7973685Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2023-01-11T20:43:46.7993044Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T20:43:46.8025715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T20:43:46.8039559Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T20:43:46.8072738Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T20:43:46.8087097Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T20:43:46.8119726Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-01-11T20:43:46.8139381Z Entering 'third_party/pocketfft'
2023-01-11T20:43:46.8172196Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2023-01-11T20:43:46.8187358Z Entering 'third_party/protobuf'
2023-01-11T20:43:46.8220427Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2023-01-11T20:43:46.8239045Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T20:43:46.8273527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T20:43:46.8287632Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T20:43:46.8322014Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2023-01-11T20:43:46.8337727Z Entering 'third_party/psimd'
2023-01-11T20:43:46.8370349Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2023-01-11T20:43:46.8385751Z Entering 'third_party/pthreadpool'
2023-01-11T20:43:46.8418266Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2023-01-11T20:43:46.8433885Z Entering 'third_party/pybind11'
2023-01-11T20:43:46.8464729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T20:43:46.8479399Z Entering 'third_party/python-enum'
2023-01-11T20:43:46.8512899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2023-01-11T20:43:46.8527758Z Entering 'third_party/python-peachpy'
2023-01-11T20:43:46.8560537Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2023-01-11T20:43:46.8574913Z Entering 'third_party/python-six'
2023-01-11T20:43:46.8607210Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2023-01-11T20:43:46.8620577Z Entering 'third_party/sleef'
2023-01-11T20:43:46.8653521Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2023-01-11T20:43:46.8668501Z Entering 'third_party/tbb'
2023-01-11T20:43:46.8699824Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2023-01-11T20:43:46.8716046Z Entering 'third_party/tensorpipe'
2023-01-11T20:43:46.8749780Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2023-01-11T20:43:46.8764426Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T20:43:46.8796312Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2023-01-11T20:43:46.8810792Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T20:43:46.8842301Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2023-01-11T20:43:46.8856748Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T20:43:46.8888890Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2023-01-11T20:43:46.8902623Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T20:43:46.8935008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T20:43:46.8948523Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T20:43:46.8983812Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-01-11T20:43:46.9000303Z Entering 'third_party/zstd'
2023-01-11T20:43:46.9032915Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2023-01-11T20:43:46.9934761Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2023-01-11T20:43:47.0202161Z Entering 'android/libs/fbjni'
2023-01-11T20:43:47.0239276Z Entering 'third_party/FP16'
2023-01-11T20:43:47.0274638Z Entering 'third_party/FXdiv'
2023-01-11T20:43:47.0311578Z Entering 'third_party/NNPACK'
2023-01-11T20:43:47.0348073Z Entering 'third_party/QNNPACK'
2023-01-11T20:43:47.0383089Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T20:43:47.0418602Z Entering 'third_party/XNNPACK'
2023-01-11T20:43:47.0468062Z Entering 'third_party/benchmark'
2023-01-11T20:43:47.0504785Z Entering 'third_party/cpuinfo'
2023-01-11T20:43:47.0542713Z Entering 'third_party/cub'
2023-01-11T20:43:47.0579213Z Entering 'third_party/cudnn_frontend'
2023-01-11T20:43:47.0621542Z Entering 'third_party/cutlass'
2023-01-11T20:43:47.0667690Z Entering 'third_party/eigen'
2023-01-11T20:43:47.0707127Z Entering 'third_party/fbgemm'
2023-01-11T20:43:47.0744351Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T20:43:47.0782161Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T20:43:47.0818299Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T20:43:47.0853235Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T20:43:47.0889008Z Entering 'third_party/flatbuffers'
2023-01-11T20:43:47.0926792Z Entering 'third_party/fmt'
2023-01-11T20:43:47.0962686Z Entering 'third_party/foxi'
2023-01-11T20:43:47.0997811Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T20:43:47.1033690Z Entering 'third_party/gloo'
2023-01-11T20:43:47.1069006Z Entering 'third_party/googletest'
2023-01-11T20:43:47.1104597Z Entering 'third_party/ideep'
2023-01-11T20:43:47.1138721Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T20:43:47.1176444Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T20:43:47.1218478Z Entering 'third_party/ios-cmake'
2023-01-11T20:43:47.1253558Z Entering 'third_party/ittapi'
2023-01-11T20:43:47.1290158Z Entering 'third_party/kineto'
2023-01-11T20:43:47.1326279Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T20:43:47.1361586Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T20:43:47.1398981Z Entering 'third_party/nccl/nccl'
2023-01-11T20:43:47.1434494Z Entering 'third_party/neon2sse'
2023-01-11T20:43:47.1469652Z Entering 'third_party/nlohmann'
2023-01-11T20:43:47.1506845Z Entering 'third_party/onnx'
2023-01-11T20:43:47.1552983Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T20:43:47.1588056Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T20:43:47.1625189Z Entering 'third_party/onnx-tensorrt'
2023-01-11T20:43:47.1660456Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T20:43:47.1699165Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T20:43:47.1734700Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T20:43:47.1769498Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T20:43:47.1808327Z Entering 'third_party/pocketfft'
2023-01-11T20:43:47.1843249Z Entering 'third_party/protobuf'
2023-01-11T20:43:47.1882285Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T20:43:47.1916895Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T20:43:47.1953605Z Entering 'third_party/psimd'
2023-01-11T20:43:47.1987439Z Entering 'third_party/pthreadpool'
2023-01-11T20:43:47.2021217Z Entering 'third_party/pybind11'
2023-01-11T20:43:47.2055702Z Entering 'third_party/python-enum'
2023-01-11T20:43:47.2088964Z Entering 'third_party/python-peachpy'
2023-01-11T20:43:47.2123405Z Entering 'third_party/python-six'
2023-01-11T20:43:47.2158035Z Entering 'third_party/sleef'
2023-01-11T20:43:47.2192366Z Entering 'third_party/tbb'
2023-01-11T20:43:47.2229157Z Entering 'third_party/tensorpipe'
2023-01-11T20:43:47.2264388Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T20:43:47.2297878Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T20:43:47.2331408Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T20:43:47.2365353Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T20:43:47.2398141Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T20:43:47.2435488Z Entering 'third_party/zstd'
2023-01-11T20:43:47.2481513Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2023-01-11T20:43:47.2735602Z Entering 'android/libs/fbjni'
2023-01-11T20:43:47.2770823Z Entering 'third_party/FP16'
2023-01-11T20:43:47.2806957Z Entering 'third_party/FXdiv'
2023-01-11T20:43:47.2841907Z Entering 'third_party/NNPACK'
2023-01-11T20:43:47.2877539Z Entering 'third_party/QNNPACK'
2023-01-11T20:43:47.2913788Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T20:43:47.2948974Z Entering 'third_party/XNNPACK'
2023-01-11T20:43:47.2995215Z Entering 'third_party/benchmark'
2023-01-11T20:43:47.3029609Z Entering 'third_party/cpuinfo'
2023-01-11T20:43:47.3064948Z Entering 'third_party/cub'
2023-01-11T20:43:47.3099832Z Entering 'third_party/cudnn_frontend'
2023-01-11T20:43:47.3139151Z Entering 'third_party/cutlass'
2023-01-11T20:43:47.3179554Z Entering 'third_party/eigen'
2023-01-11T20:43:47.3215287Z Entering 'third_party/fbgemm'
2023-01-11T20:43:47.3251216Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T20:43:47.3285589Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T20:43:47.3319633Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T20:43:47.3353181Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T20:43:47.3387622Z Entering 'third_party/flatbuffers'
2023-01-11T20:43:47.3424085Z Entering 'third_party/fmt'
2023-01-11T20:43:47.3458414Z Entering 'third_party/foxi'
2023-01-11T20:43:47.3493129Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T20:43:47.3528041Z Entering 'third_party/gloo'
2023-01-11T20:43:47.3562421Z Entering 'third_party/googletest'
2023-01-11T20:43:47.3597128Z Entering 'third_party/ideep'
2023-01-11T20:43:47.3634350Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T20:43:47.3670599Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T20:43:47.3710800Z Entering 'third_party/ios-cmake'
2023-01-11T20:43:47.3747073Z Entering 'third_party/ittapi'
2023-01-11T20:43:47.3781242Z Entering 'third_party/kineto'
2023-01-11T20:43:47.3816312Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T20:43:47.3850786Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T20:43:47.3886501Z Entering 'third_party/nccl/nccl'
2023-01-11T20:43:47.3922248Z Entering 'third_party/neon2sse'
2023-01-11T20:43:47.3957164Z Entering 'third_party/nlohmann'
2023-01-11T20:43:47.3993165Z Entering 'third_party/onnx'
2023-01-11T20:43:47.4040270Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T20:43:47.4075157Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T20:43:47.4113104Z Entering 'third_party/onnx-tensorrt'
2023-01-11T20:43:47.4148196Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T20:43:47.4188241Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T20:43:47.4222709Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T20:43:47.4256695Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T20:43:47.4296111Z Entering 'third_party/pocketfft'
2023-01-11T20:43:47.4331138Z Entering 'third_party/protobuf'
2023-01-11T20:43:47.4371173Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T20:43:47.4413533Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T20:43:47.4446894Z Entering 'third_party/psimd'
2023-01-11T20:43:47.4482052Z Entering 'third_party/pthreadpool'
2023-01-11T20:43:47.4517635Z Entering 'third_party/pybind11'
2023-01-11T20:43:47.4554028Z Entering 'third_party/python-enum'
2023-01-11T20:43:47.4589018Z Entering 'third_party/python-peachpy'
2023-01-11T20:43:47.4624576Z Entering 'third_party/python-six'
2023-01-11T20:43:47.4659471Z Entering 'third_party/sleef'
2023-01-11T20:43:47.4694254Z Entering 'third_party/tbb'
2023-01-11T20:43:47.4732588Z Entering 'third_party/tensorpipe'
2023-01-11T20:43:47.4767971Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T20:43:47.4804791Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T20:43:47.4840374Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T20:43:47.4875571Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T20:43:47.4911024Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T20:43:47.4948672Z Entering 'third_party/zstd'
2023-01-11T20:43:47.4993396Z ##[endgroup]
2023-01-11T20:43:47.5035930Z [command]/usr/bin/git log -1 --format='%H'
2023-01-11T20:43:47.5064856Z '8419ddda87c8a47eacc63b54bc7ec98c1f27c26e'
2023-01-11T20:43:47.5188388Z Prepare all required actions
2023-01-11T20:43:47.5214102Z ##[group]Run ./.github/actions/setup-linux
2023-01-11T20:43:47.5214307Z env:
2023-01-11T20:43:47.5214482Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:47.5240358Z ##[endgroup]
2023-01-11T20:43:47.5256382Z ##[group]Run set -euo pipefail
2023-01-11T20:43:47.5256618Z set -euo pipefail
2023-01-11T20:43:47.5256815Z function get_ec2_metadata() {
2023-01-11T20:43:47.5257060Z   # Pulled from instance metadata endpoint for EC2
2023-01-11T20:43:47.5257412Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2023-01-11T20:43:47.5257705Z   category=$1
2023-01-11T20:43:47.5257943Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2023-01-11T20:43:47.5258156Z }
2023-01-11T20:43:47.5258340Z echo "ami-id: $(get_ec2_metadata ami-id)"
2023-01-11T20:43:47.5258611Z echo "instance-id: $(get_ec2_metadata instance-id)"
2023-01-11T20:43:47.5258883Z echo "instance-type: $(get_ec2_metadata instance-type)"
2023-01-11T20:43:47.5259116Z echo "system info $(uname -a)"
2023-01-11T20:43:47.5270665Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:43:47.5270885Z env:
2023-01-11T20:43:47.5271052Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:47.5271231Z ##[endgroup]
2023-01-11T20:43:47.5348576Z ami-id: ami-096198a0bccc6bad4
2023-01-11T20:43:47.5396811Z instance-id: i-03e06aa264bd8cb8d
2023-01-11T20:43:47.5444261Z instance-type: c5.2xlarge
2023-01-11T20:43:47.5451124Z system info Linux ip-10-0-3-131.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2023-01-11T20:43:47.5468572Z ##[group]Run if systemctl is-active --quiet docker; then
2023-01-11T20:43:47.5468857Z if systemctl is-active --quiet docker; then
2023-01-11T20:43:47.5469102Z     echo "Docker daemon is running...";
2023-01-11T20:43:47.5469313Z else
2023-01-11T20:43:47.5469533Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2023-01-11T20:43:47.5469760Z fi
2023-01-11T20:43:47.5480535Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:43:47.5480737Z env:
2023-01-11T20:43:47.5480911Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:47.5481092Z ##[endgroup]
2023-01-11T20:43:47.5521672Z Docker daemon is running...
2023-01-11T20:43:47.5538486Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2023-01-11T20:43:47.5538848Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2023-01-11T20:43:47.5539137Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T20:43:47.5539522Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2023-01-11T20:43:47.5539872Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2023-01-11T20:43:47.5550639Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:43:47.5550865Z env:
2023-01-11T20:43:47.5551042Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:47.5551226Z   AWS_RETRY_MODE: standard
2023-01-11T20:43:47.5551416Z   AWS_MAX_ATTEMPTS: 5
2023-01-11T20:43:47.5551608Z   AWS_DEFAULT_REGION: us-east-1
2023-01-11T20:43:47.5551795Z ##[endgroup]
2023-01-11T20:43:48.3188723Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2023-01-11T20:43:48.3189106Z Configure a credential helper to remove this warning. See
2023-01-11T20:43:48.3189770Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2023-01-11T20:43:48.3190308Z 
2023-01-11T20:43:48.3190666Z Login Succeeded
2023-01-11T20:43:48.3231229Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T20:43:48.3231740Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T20:43:48.3232309Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T20:43:48.3243753Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:43:48.3244074Z env:
2023-01-11T20:43:48.3244252Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:48.3244426Z ##[endgroup]
2023-01-11T20:43:48.3328104Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2023-01-11T20:43:48.3328358Z with:
2023-01-11T20:43:48.3328683Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:43:48.3329012Z env:
2023-01-11T20:43:48.3329197Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:48.3329383Z ##[endgroup]
2023-01-11T20:43:48.3343547Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T20:43:48.3343824Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T20:43:48.3344090Z # ignore output since only exit code is used for conditional
2023-01-11T20:43:48.3344371Z # only pull docker image if it's not available locally
2023-01-11T20:43:48.3344666Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2023-01-11T20:43:48.3344965Z   retry docker pull "${DOCKER_IMAGE}"
2023-01-11T20:43:48.3345161Z fi
2023-01-11T20:43:48.3356349Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:43:48.3356570Z env:
2023-01-11T20:43:48.3356752Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:48.3357105Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:43:48.3357451Z ##[endgroup]
2023-01-11T20:43:48.5579976Z fd224c2e6c79d7fdec6408da598bf52bc5b201dd: Pulling from pytorch/pytorch-linux-focal-py3.7-gcc7
2023-01-11T20:43:48.5580586Z 846c0b181fff: Pulling fs layer
2023-01-11T20:43:48.5580918Z 0698304963f1: Pulling fs layer
2023-01-11T20:43:48.5581212Z 77ce1d4f36bf: Pulling fs layer
2023-01-11T20:43:48.5581480Z d39938d48fa2: Pulling fs layer
2023-01-11T20:43:48.5581916Z ae4dac5a8919: Pulling fs layer
2023-01-11T20:43:48.5582265Z 4ff68f5f773b: Pulling fs layer
2023-01-11T20:43:48.5582621Z 47642f12aa0f: Pulling fs layer
2023-01-11T20:43:48.5582923Z 3c02291730dc: Pulling fs layer
2023-01-11T20:43:48.5583114Z c7165c4f5e23: Pulling fs layer
2023-01-11T20:43:48.5583294Z 8357faec0a0e: Pulling fs layer
2023-01-11T20:43:48.5583476Z ae4dac5a8919: Waiting
2023-01-11T20:43:48.5583669Z 42fee15d0ad3: Pulling fs layer
2023-01-11T20:43:48.5583851Z 246016b9e8c8: Pulling fs layer
2023-01-11T20:43:48.5585356Z 4ff68f5f773b: Waiting
2023-01-11T20:43:48.5585685Z 25be21861eed: Pulling fs layer
2023-01-11T20:43:48.5585956Z 21ac276b84eb: Pulling fs layer
2023-01-11T20:43:48.5586245Z bd695a285c41: Pulling fs layer
2023-01-11T20:43:48.5586497Z 9af030ba5896: Pulling fs layer
2023-01-11T20:43:48.5586781Z 0ab7fbe28a03: Pulling fs layer
2023-01-11T20:43:48.5587075Z f5dba516d75a: Pulling fs layer
2023-01-11T20:43:48.5587352Z 944388bd6de3: Pulling fs layer
2023-01-11T20:43:48.5587626Z 32233288f470: Pulling fs layer
2023-01-11T20:43:48.5587904Z 4716e681e05e: Pulling fs layer
2023-01-11T20:43:48.5588165Z 47642f12aa0f: Waiting
2023-01-11T20:43:48.5593218Z 07008cd698a0: Pulling fs layer
2023-01-11T20:43:48.5593731Z 27f046e694b1: Pulling fs layer
2023-01-11T20:43:48.5594036Z 3c02291730dc: Waiting
2023-01-11T20:43:48.5596737Z 95f903a2e692: Pulling fs layer
2023-01-11T20:43:48.5597047Z 1d465abcef1e: Pulling fs layer
2023-01-11T20:43:48.5597356Z c7165c4f5e23: Waiting
2023-01-11T20:43:48.5597679Z 32fdce95f2af: Pulling fs layer
2023-01-11T20:43:48.5598049Z 8357faec0a0e: Waiting
2023-01-11T20:43:48.5598246Z 38f17f1788cc: Pulling fs layer
2023-01-11T20:43:48.5598445Z fa2049e24ad0: Pulling fs layer
2023-01-11T20:43:48.5598619Z 42fee15d0ad3: Waiting
2023-01-11T20:43:48.5598808Z 070d94af58e3: Pulling fs layer
2023-01-11T20:43:48.5598994Z 25be21861eed: Waiting
2023-01-11T20:43:48.5599157Z 246016b9e8c8: Waiting
2023-01-11T20:43:48.5599338Z 9ff218827ef0: Pulling fs layer
2023-01-11T20:43:48.5599535Z 0effc56e2dd9: Pulling fs layer
2023-01-11T20:43:48.5599712Z 21ac276b84eb: Waiting
2023-01-11T20:43:48.5599898Z bd453ee44627: Pulling fs layer
2023-01-11T20:43:48.5600079Z 32233288f470: Waiting
2023-01-11T20:43:48.5600332Z e999c29eb8b9: Pulling fs layer
2023-01-11T20:43:48.5600523Z f5dba516d75a: Waiting
2023-01-11T20:43:48.5600698Z bd695a285c41: Waiting
2023-01-11T20:43:48.5600868Z 566e975723c3: Pulling fs layer
2023-01-11T20:43:48.5601051Z 4716e681e05e: Waiting
2023-01-11T20:43:48.5601223Z 944388bd6de3: Waiting
2023-01-11T20:43:48.5601452Z 1072a14c5615: Pulling fs layer
2023-01-11T20:43:48.5601653Z 9af030ba5896: Waiting
2023-01-11T20:43:48.5601841Z 9ae70c90b543: Pulling fs layer
2023-01-11T20:43:48.5602024Z 27f046e694b1: Waiting
2023-01-11T20:43:48.5602186Z 0ab7fbe28a03: Waiting
2023-01-11T20:43:48.5602355Z 95f903a2e692: Waiting
2023-01-11T20:43:48.5602535Z e6f8bf185266: Pulling fs layer
2023-01-11T20:43:48.5602702Z d39938d48fa2: Waiting
2023-01-11T20:43:48.5602884Z 01ff2898de6c: Pulling fs layer
2023-01-11T20:43:48.5603112Z 1d465abcef1e: Waiting
2023-01-11T20:43:48.5603355Z bd453ee44627: Waiting
2023-01-11T20:43:48.5603626Z 98f07e351fe7: Pulling fs layer
2023-01-11T20:43:48.5604004Z 38f17f1788cc: Waiting
2023-01-11T20:43:48.5604302Z 32fdce95f2af: Waiting
2023-01-11T20:43:48.5604632Z e426b18803bb: Pulling fs layer
2023-01-11T20:43:48.5604921Z 0effc56e2dd9: Waiting
2023-01-11T20:43:48.5605194Z e999c29eb8b9: Waiting
2023-01-11T20:43:48.5605498Z da3dd65b2531: Pulling fs layer
2023-01-11T20:43:48.5605786Z 070d94af58e3: Waiting
2023-01-11T20:43:48.5606053Z fa2049e24ad0: Waiting
2023-01-11T20:43:48.5606337Z 1072a14c5615: Waiting
2023-01-11T20:43:48.5606606Z 5c3ec8735bee: Pulling fs layer
2023-01-11T20:43:48.5606899Z 9ff218827ef0: Waiting
2023-01-11T20:43:48.5607196Z b616c89406f0: Pulling fs layer
2023-01-11T20:43:48.5607500Z e6f8bf185266: Waiting
2023-01-11T20:43:48.5607784Z f48f88ff2185: Pulling fs layer
2023-01-11T20:43:48.5608099Z 46e0cb4c14f2: Pulling fs layer
2023-01-11T20:43:48.5608424Z 300515e4431f: Pulling fs layer
2023-01-11T20:43:48.5610665Z 01ff2898de6c: Waiting
2023-01-11T20:43:48.5611040Z 9df79d1a6e27: Pulling fs layer
2023-01-11T20:43:48.5611376Z f48f88ff2185: Waiting
2023-01-11T20:43:48.5611688Z 5c3ec8735bee: Waiting
2023-01-11T20:43:48.5612002Z 3cda6364e15e: Pulling fs layer
2023-01-11T20:43:48.5612352Z 7fc4121e1952: Pulling fs layer
2023-01-11T20:43:48.5612694Z 676dcfb6d5c2: Pulling fs layer
2023-01-11T20:43:48.5612974Z 130a71e99e46: Pulling fs layer
2023-01-11T20:43:48.5613167Z 9df79d1a6e27: Waiting
2023-01-11T20:43:48.5613354Z 932461d264d4: Pulling fs layer
2023-01-11T20:43:48.5613595Z 6cce994e24e8: Pulling fs layer
2023-01-11T20:43:48.5614615Z b37b5d679247: Pulling fs layer
2023-01-11T20:43:48.5614940Z 3cda6364e15e: Waiting
2023-01-11T20:43:48.5615247Z 86ad3a500f17: Pulling fs layer
2023-01-11T20:43:48.5615554Z 93e8483d7d76: Pulling fs layer
2023-01-11T20:43:48.5615865Z 7fc4121e1952: Waiting
2023-01-11T20:43:48.5616148Z 676dcfb6d5c2: Waiting
2023-01-11T20:43:48.5616421Z ad1ecb94a573: Pulling fs layer
2023-01-11T20:43:48.5616741Z a7f9c3a7c21d: Pulling fs layer
2023-01-11T20:43:48.5617027Z 130a71e99e46: Waiting
2023-01-11T20:43:48.5617293Z 6cce994e24e8: Waiting
2023-01-11T20:43:48.5617588Z 932461d264d4: Waiting
2023-01-11T20:43:48.5617893Z 86ad3a500f17: Waiting
2023-01-11T20:43:48.5618181Z a7f9c3a7c21d: Waiting
2023-01-11T20:43:48.5618478Z b37b5d679247: Waiting
2023-01-11T20:43:48.5618753Z 93e8483d7d76: Waiting
2023-01-11T20:43:48.6252300Z 0698304963f1: Verifying Checksum
2023-01-11T20:43:48.6252664Z 0698304963f1: Download complete
2023-01-11T20:43:48.6969563Z d39938d48fa2: Verifying Checksum
2023-01-11T20:43:48.6969843Z d39938d48fa2: Download complete
2023-01-11T20:43:48.7636259Z ae4dac5a8919: Download complete
2023-01-11T20:43:48.8356553Z 4ff68f5f773b: Verifying Checksum
2023-01-11T20:43:48.8356937Z 4ff68f5f773b: Download complete
2023-01-11T20:43:48.8958571Z 846c0b181fff: Download complete
2023-01-11T20:43:48.9085244Z 47642f12aa0f: Download complete
2023-01-11T20:43:48.9814483Z 3c02291730dc: Download complete
2023-01-11T20:43:48.9821321Z c7165c4f5e23: Verifying Checksum
2023-01-11T20:43:48.9821918Z c7165c4f5e23: Download complete
2023-01-11T20:43:49.0585917Z 8357faec0a0e: Verifying Checksum
2023-01-11T20:43:49.0586511Z 8357faec0a0e: Download complete
2023-01-11T20:43:49.1428441Z 246016b9e8c8: Download complete
2023-01-11T20:43:49.2056298Z 25be21861eed: Download complete
2023-01-11T20:43:49.6227065Z 846c0b181fff: Pull complete
2023-01-11T20:43:49.7232482Z 0698304963f1: Pull complete
2023-01-11T20:43:49.8623986Z 42fee15d0ad3: Verifying Checksum
2023-01-11T20:43:49.8624412Z 42fee15d0ad3: Download complete
2023-01-11T20:43:49.9325667Z bd695a285c41: Verifying Checksum
2023-01-11T20:43:49.9326057Z bd695a285c41: Download complete
2023-01-11T20:43:50.0106599Z 9af030ba5896: Verifying Checksum
2023-01-11T20:43:50.0107034Z 9af030ba5896: Download complete
2023-01-11T20:43:50.5094580Z 0ab7fbe28a03: Verifying Checksum
2023-01-11T20:43:50.5094985Z 0ab7fbe28a03: Download complete
2023-01-11T20:43:50.6006529Z f5dba516d75a: Verifying Checksum
2023-01-11T20:43:50.6006965Z f5dba516d75a: Download complete
2023-01-11T20:43:50.7137829Z 944388bd6de3: Verifying Checksum
2023-01-11T20:43:50.7138174Z 944388bd6de3: Download complete
2023-01-11T20:43:50.8095933Z 32233288f470: Verifying Checksum
2023-01-11T20:43:50.8096319Z 32233288f470: Download complete
2023-01-11T20:43:51.7004984Z 77ce1d4f36bf: Verifying Checksum
2023-01-11T20:43:51.7005385Z 77ce1d4f36bf: Download complete
2023-01-11T20:43:51.7770314Z 07008cd698a0: Download complete
2023-01-11T20:43:51.8556958Z 27f046e694b1: Verifying Checksum
2023-01-11T20:43:51.8557413Z 27f046e694b1: Download complete
2023-01-11T20:43:51.9312426Z 95f903a2e692: Download complete
2023-01-11T20:43:52.8647328Z 1d465abcef1e: Verifying Checksum
2023-01-11T20:43:52.8647707Z 1d465abcef1e: Download complete
2023-01-11T20:43:52.9371589Z 32fdce95f2af: Verifying Checksum
2023-01-11T20:43:52.9371894Z 32fdce95f2af: Download complete
2023-01-11T20:43:53.0171167Z 38f17f1788cc: Verifying Checksum
2023-01-11T20:43:53.0171538Z 38f17f1788cc: Download complete
2023-01-11T20:43:53.1160204Z fa2049e24ad0: Verifying Checksum
2023-01-11T20:43:53.1160480Z fa2049e24ad0: Download complete
2023-01-11T20:43:53.1928745Z 070d94af58e3: Verifying Checksum
2023-01-11T20:43:53.1929759Z 070d94af58e3: Download complete
2023-01-11T20:43:53.2751238Z 9ff218827ef0: Download complete
2023-01-11T20:43:54.6342026Z 4716e681e05e: Download complete
2023-01-11T20:43:54.7013587Z bd453ee44627: Verifying Checksum
2023-01-11T20:43:54.7013989Z bd453ee44627: Download complete
2023-01-11T20:43:54.7812016Z e999c29eb8b9: Verifying Checksum
2023-01-11T20:43:54.8617557Z 566e975723c3: Verifying Checksum
2023-01-11T20:43:54.8617972Z 566e975723c3: Download complete
2023-01-11T20:43:54.9681057Z 1072a14c5615: Download complete
2023-01-11T20:43:55.0373963Z 9ae70c90b543: Verifying Checksum
2023-01-11T20:43:55.0374448Z 9ae70c90b543: Download complete
2023-01-11T20:43:55.1044689Z e6f8bf185266: Verifying Checksum
2023-01-11T20:43:55.1888890Z e6f8bf185266: Download complete
2023-01-11T20:43:55.1889336Z 01ff2898de6c: Verifying Checksum
2023-01-11T20:43:55.1889695Z 01ff2898de6c: Download complete
2023-01-11T20:43:55.2585302Z 98f07e351fe7: Verifying Checksum
2023-01-11T20:43:55.2585739Z 98f07e351fe7: Download complete
2023-01-11T20:43:55.3254490Z e426b18803bb: Verifying Checksum
2023-01-11T20:43:55.3254895Z e426b18803bb: Download complete
2023-01-11T20:43:55.4130119Z da3dd65b2531: Verifying Checksum
2023-01-11T20:43:55.4130540Z da3dd65b2531: Download complete
2023-01-11T20:43:55.4882587Z 5c3ec8735bee: Download complete
2023-01-11T20:43:55.5232072Z 0effc56e2dd9: Verifying Checksum
2023-01-11T20:43:55.5232617Z 0effc56e2dd9: Download complete
2023-01-11T20:43:55.5704407Z b616c89406f0: Download complete
2023-01-11T20:43:55.6506626Z 46e0cb4c14f2: Verifying Checksum
2023-01-11T20:43:55.6507014Z 46e0cb4c14f2: Download complete
2023-01-11T20:43:55.7416969Z 300515e4431f: Verifying Checksum
2023-01-11T20:43:55.7417318Z 300515e4431f: Download complete
2023-01-11T20:43:55.8257486Z 9df79d1a6e27: Download complete
2023-01-11T20:43:55.8960254Z 3cda6364e15e: Verifying Checksum
2023-01-11T20:43:55.8961134Z 3cda6364e15e: Download complete
2023-01-11T20:43:56.0105057Z f48f88ff2185: Verifying Checksum
2023-01-11T20:43:56.0105478Z f48f88ff2185: Download complete
2023-01-11T20:43:56.0769182Z 676dcfb6d5c2: Verifying Checksum
2023-01-11T20:43:56.0769592Z 676dcfb6d5c2: Download complete
2023-01-11T20:43:56.1415865Z 7fc4121e1952: Verifying Checksum
2023-01-11T20:43:56.1416350Z 7fc4121e1952: Download complete
2023-01-11T20:43:56.1613559Z 130a71e99e46: Download complete
2023-01-11T20:43:56.2322694Z 6cce994e24e8: Verifying Checksum
2023-01-11T20:43:56.3132185Z b37b5d679247: Download complete
2023-01-11T20:43:56.4553574Z 86ad3a500f17: Verifying Checksum
2023-01-11T20:43:56.4554008Z 86ad3a500f17: Download complete
2023-01-11T20:43:56.5248921Z 93e8483d7d76: Download complete
2023-01-11T20:43:56.6155543Z ad1ecb94a573: Verifying Checksum
2023-01-11T20:43:56.6155953Z ad1ecb94a573: Download complete
2023-01-11T20:43:57.2191816Z a7f9c3a7c21d: Verifying Checksum
2023-01-11T20:43:57.2192249Z a7f9c3a7c21d: Download complete
2023-01-11T20:43:58.3052764Z 77ce1d4f36bf: Pull complete
2023-01-11T20:43:58.5455881Z d39938d48fa2: Pull complete
2023-01-11T20:43:58.7814862Z ae4dac5a8919: Pull complete
2023-01-11T20:43:58.9031283Z 932461d264d4: Download complete
2023-01-11T20:43:59.0343398Z 4ff68f5f773b: Pull complete
2023-01-11T20:43:59.2896250Z 47642f12aa0f: Pull complete
2023-01-11T20:43:59.5240463Z 3c02291730dc: Pull complete
2023-01-11T20:43:59.7804684Z c7165c4f5e23: Pull complete
2023-01-11T20:43:59.9652622Z 8357faec0a0e: Pull complete
2023-01-11T20:44:01.2777812Z 21ac276b84eb: Download complete
2023-01-11T20:44:01.7926804Z 42fee15d0ad3: Pull complete
2023-01-11T20:44:02.0641988Z 246016b9e8c8: Pull complete
2023-01-11T20:44:02.2951095Z 25be21861eed: Pull complete
2023-01-11T20:44:22.6340246Z 21ac276b84eb: Pull complete
2023-01-11T20:44:22.9636219Z bd695a285c41: Pull complete
2023-01-11T20:44:23.1970619Z 9af030ba5896: Pull complete
2023-01-11T20:44:23.9844870Z 0ab7fbe28a03: Pull complete
2023-01-11T20:44:24.1008921Z f5dba516d75a: Pull complete
2023-01-11T20:44:24.2974902Z 944388bd6de3: Pull complete
2023-01-11T20:44:24.4679506Z 32233288f470: Pull complete
2023-01-11T20:44:24.5724084Z 4716e681e05e: Pull complete
2023-01-11T20:44:24.6847103Z 07008cd698a0: Pull complete
2023-01-11T20:44:24.7885591Z 27f046e694b1: Pull complete
2023-01-11T20:44:24.8846204Z 95f903a2e692: Pull complete
2023-01-11T20:44:26.7622173Z 1d465abcef1e: Pull complete
2023-01-11T20:44:26.9074698Z 32fdce95f2af: Pull complete
2023-01-11T20:44:27.1154392Z 38f17f1788cc: Pull complete
2023-01-11T20:44:27.2688935Z fa2049e24ad0: Pull complete
2023-01-11T20:44:27.4805820Z 070d94af58e3: Pull complete
2023-01-11T20:44:27.7205240Z 9ff218827ef0: Pull complete
2023-01-11T20:44:32.3470013Z 0effc56e2dd9: Pull complete
2023-01-11T20:44:32.5683100Z bd453ee44627: Pull complete
2023-01-11T20:44:32.7432969Z e999c29eb8b9: Pull complete
2023-01-11T20:44:32.9050809Z 566e975723c3: Pull complete
2023-01-11T20:44:33.0650795Z 1072a14c5615: Pull complete
2023-01-11T20:44:33.3037680Z 9ae70c90b543: Pull complete
2023-01-11T20:44:33.4933330Z e6f8bf185266: Pull complete
2023-01-11T20:44:33.5930492Z 01ff2898de6c: Pull complete
2023-01-11T20:44:33.6795344Z 98f07e351fe7: Pull complete
2023-01-11T20:44:33.7854692Z e426b18803bb: Pull complete
2023-01-11T20:44:33.8698827Z da3dd65b2531: Pull complete
2023-01-11T20:44:33.9604440Z 5c3ec8735bee: Pull complete
2023-01-11T20:44:34.0776740Z b616c89406f0: Pull complete
2023-01-11T20:44:35.3440188Z f48f88ff2185: Pull complete
2023-01-11T20:44:35.5727467Z 46e0cb4c14f2: Pull complete
2023-01-11T20:44:35.8139986Z 300515e4431f: Pull complete
2023-01-11T20:44:36.0062157Z 9df79d1a6e27: Pull complete
2023-01-11T20:44:36.2603866Z 3cda6364e15e: Pull complete
2023-01-11T20:44:36.7329518Z 7fc4121e1952: Pull complete
2023-01-11T20:44:36.9712232Z 676dcfb6d5c2: Pull complete
2023-01-11T20:44:37.2158132Z 130a71e99e46: Pull complete
2023-01-11T20:44:41.0815604Z 932461d264d4: Pull complete
2023-01-11T20:44:41.3319320Z 6cce994e24e8: Pull complete
2023-01-11T20:44:41.5558276Z b37b5d679247: Pull complete
2023-01-11T20:44:41.7681150Z 86ad3a500f17: Pull complete
2023-01-11T20:44:42.0162994Z 93e8483d7d76: Pull complete
2023-01-11T20:44:42.2378077Z ad1ecb94a573: Pull complete
2023-01-11T20:44:43.7496144Z a7f9c3a7c21d: Pull complete
2023-01-11T20:44:43.7610990Z Digest: sha256:d201d5d219c1681863b90b8d50e8b2382dd0a533dc69c772527c8255937144a4
2023-01-11T20:44:43.7645707Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:44:43.7677576Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:44:43.7727021Z ##[group]Run python3 -m pip install psutil==5.9.1
2023-01-11T20:44:43.7727301Z python3 -m pip install psutil==5.9.1
2023-01-11T20:44:43.7727524Z python3 -m pip install pynvml==11.4.1
2023-01-11T20:44:43.7727779Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2023-01-11T20:44:43.7728056Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2023-01-11T20:44:43.7739364Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:44:43.7739583Z env:
2023-01-11T20:44:43.7739758Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:43.7739928Z ##[endgroup]
2023-01-11T20:44:43.9918957Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T20:44:44.0104829Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1)
2023-01-11T20:44:44.4332676Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T20:44:44.4515125Z Requirement already satisfied: pynvml==11.4.1 in /home/ec2-user/.local/lib/python3.7/site-packages (11.4.1)
2023-01-11T20:44:44.6625228Z Prepare all required actions
2023-01-11T20:44:44.6625487Z Getting action download info
2023-01-11T20:44:44.8230284Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2023-01-11T20:44:44.9868365Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a)
2023-01-11T20:44:45.1274961Z ##[group]Run ./.github/actions/download-build-artifacts
2023-01-11T20:44:45.1275172Z with:
2023-01-11T20:44:45.1275356Z   name: linux-focal-py3.7-gcc7
2023-01-11T20:44:45.1275534Z env:
2023-01-11T20:44:45.1275691Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:45.1275877Z ##[endgroup]
2023-01-11T20:44:45.1300777Z ##[group]Run seemethere/download-artifact-s3@v4
2023-01-11T20:44:45.1300986Z with:
2023-01-11T20:44:45.1301158Z   name: linux-focal-py3.7-gcc7
2023-01-11T20:44:45.1301342Z   s3-bucket: gha-artifacts
2023-01-11T20:44:45.1301516Z   region: us-east-1
2023-01-11T20:44:45.1301815Z env:
2023-01-11T20:44:45.1302016Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:45.1302189Z ##[endgroup]
2023-01-11T20:44:45.5405822Z Found 1 objects with prefix pytorch/pytorch/3896099317/linux-focal-py3.7-gcc7/
2023-01-11T20:44:45.5406278Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-01-11T20:44:48.6726694Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-01-11T20:44:48.6728281Z 
2023-01-11T20:44:48.6745005Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2023-01-11T20:44:48.6754020Z Artifact download has finished successfully
2023-01-11T20:44:48.6840405Z ##[group]Run unzip -o artifacts.zip
2023-01-11T20:44:48.6840624Z unzip -o artifacts.zip
2023-01-11T20:44:48.6852164Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:44:48.6852394Z env:
2023-01-11T20:44:48.6852557Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:48.6852748Z ##[endgroup]
2023-01-11T20:44:48.6889133Z Archive:  artifacts.zip
2023-01-11T20:44:48.6890432Z    creating: dist/
2023-01-11T20:44:49.4587070Z   inflating: dist/torch-2.0.0a0+git8419ddd-cp37-cp37m-linux_x86_64.whl  
2023-01-11T20:44:49.4587442Z    creating: build/custom_test_artifacts/
2023-01-11T20:44:49.4587746Z    creating: build/custom_test_artifacts/custom-op-build/
2023-01-11T20:44:49.4588103Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2023-01-11T20:44:49.4589628Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2023-01-11T20:44:49.4590071Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2023-01-11T20:44:49.4590571Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T20:44:49.4591000Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T20:44:49.4591416Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T20:44:49.4593196Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T20:44:49.4594271Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T20:44:49.4594984Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T20:44:49.4595697Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T20:44:49.4597116Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T20:44:49.4598403Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T20:44:49.4599957Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T20:44:49.4600767Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T20:44:49.4601757Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T20:44:49.4602540Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T20:44:49.4603248Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2023-01-11T20:44:49.4604011Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2023-01-11T20:44:49.4604727Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2023-01-11T20:44:49.4605362Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2023-01-11T20:44:49.4605816Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2023-01-11T20:44:49.4606348Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2023-01-11T20:44:49.4607013Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2023-01-11T20:44:49.4607441Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2023-01-11T20:44:49.4607886Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2023-01-11T20:44:49.4608464Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2023-01-11T20:44:49.4609141Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2023-01-11T20:44:49.4609723Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2023-01-11T20:44:49.4610167Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2023-01-11T20:44:49.4624559Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2023-01-11T20:44:49.4734776Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2023-01-11T20:44:49.4735242Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2023-01-11T20:44:49.4735696Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2023-01-11T20:44:49.4736180Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2023-01-11T20:44:49.4736638Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2023-01-11T20:44:49.4737083Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2023-01-11T20:44:49.4737543Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2023-01-11T20:44:49.4737996Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2023-01-11T20:44:49.4738438Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2023-01-11T20:44:49.4738895Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2023-01-11T20:44:49.4739359Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2023-01-11T20:44:49.4756121Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2023-01-11T20:44:49.4823006Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2023-01-11T20:44:49.4823499Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T20:44:49.4824088Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T20:44:49.4824522Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2023-01-11T20:44:49.4825275Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2023-01-11T20:44:49.4826168Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2023-01-11T20:44:49.4827824Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2023-01-11T20:44:49.4828729Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2023-01-11T20:44:49.4829971Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2023-01-11T20:44:49.4903346Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2023-01-11T20:44:49.4953764Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2023-01-11T20:44:49.4954417Z    creating: build/custom_test_artifacts/jit-hook-build/
2023-01-11T20:44:49.4955057Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2023-01-11T20:44:49.4957479Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2023-01-11T20:44:49.4958206Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2023-01-11T20:44:49.4958961Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T20:44:49.4959713Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T20:44:49.4960421Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T20:44:49.4961409Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T20:44:49.4963173Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T20:44:49.4964022Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T20:44:49.4964775Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T20:44:49.4966742Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T20:44:49.4968355Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T20:44:49.4970495Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T20:44:49.4971329Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T20:44:49.4972855Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T20:44:49.4973822Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T20:44:49.4974578Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2023-01-11T20:44:49.4975288Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2023-01-11T20:44:49.4976039Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2023-01-11T20:44:49.4976776Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2023-01-11T20:44:49.4977595Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2023-01-11T20:44:49.4978418Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2023-01-11T20:44:49.4979237Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2023-01-11T20:44:49.4980063Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2023-01-11T20:44:49.4980888Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2023-01-11T20:44:49.4982083Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2023-01-11T20:44:49.4982928Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2023-01-11T20:44:49.4983715Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2023-01-11T20:44:49.4984506Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2023-01-11T20:44:49.4998748Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2023-01-11T20:44:49.5056700Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2023-01-11T20:44:49.5057619Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T20:44:49.5058346Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T20:44:49.5059030Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2023-01-11T20:44:49.5059910Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2023-01-11T20:44:49.5060930Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2023-01-11T20:44:49.5063141Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2023-01-11T20:44:49.5065308Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2023-01-11T20:44:49.5066348Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2023-01-11T20:44:49.5113054Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2023-01-11T20:44:49.5113728Z    creating: build/custom_test_artifacts/custom-backend-build/
2023-01-11T20:44:49.5114352Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2023-01-11T20:44:49.5117586Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2023-01-11T20:44:49.5118334Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2023-01-11T20:44:49.5119153Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T20:44:49.5119940Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T20:44:49.5120719Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T20:44:49.5121595Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T20:44:49.5124361Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T20:44:49.5125169Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T20:44:49.5125980Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T20:44:49.5128517Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T20:44:49.5129999Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T20:44:49.5132155Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T20:44:49.5133049Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T20:44:49.5134478Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T20:44:49.5135484Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T20:44:49.5136265Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2023-01-11T20:44:49.5137067Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2023-01-11T20:44:49.5137921Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2023-01-11T20:44:49.5138395Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2023-01-11T20:44:49.5138954Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2023-01-11T20:44:49.5140520Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2023-01-11T20:44:49.5141002Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2023-01-11T20:44:49.5142034Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2023-01-11T20:44:49.5142943Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2023-01-11T20:44:49.5143839Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2023-01-11T20:44:49.5144741Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2023-01-11T20:44:49.5145634Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2023-01-11T20:44:49.5146502Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2023-01-11T20:44:49.5149798Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2023-01-11T20:44:49.5274927Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2023-01-11T20:44:49.5275849Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2023-01-11T20:44:49.5276770Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2023-01-11T20:44:49.5277709Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2023-01-11T20:44:49.5278620Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2023-01-11T20:44:49.5279516Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2023-01-11T20:44:49.5280412Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2023-01-11T20:44:49.5281310Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2023-01-11T20:44:49.5282206Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2023-01-11T20:44:49.5283088Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2023-01-11T20:44:49.5284029Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2023-01-11T20:44:49.5297610Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2023-01-11T20:44:49.5345331Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2023-01-11T20:44:49.5346318Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T20:44:49.5347184Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T20:44:49.5348018Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2023-01-11T20:44:49.5348776Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2023-01-11T20:44:49.5349657Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2023-01-11T20:44:49.5350423Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2023-01-11T20:44:49.5351302Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2023-01-11T20:44:49.5351998Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2023-01-11T20:44:49.5447956Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2023-01-11T20:44:49.5484545Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2023-01-11T20:44:49.5484833Z    creating: build/lib/
2023-01-11T20:44:49.5485387Z   inflating: build/lib/libclog.a     
2023-01-11T20:44:49.5539276Z   inflating: build/lib/libgtest.a    
2023-01-11T20:44:49.5547397Z   inflating: build/lib/libpthreadpool.a  
2023-01-11T20:44:49.5554372Z   inflating: build/lib/libittnotify.a  
2023-01-11T20:44:49.5629483Z   inflating: build/lib/libbenchmark.a  
2023-01-11T20:44:49.5711539Z   inflating: build/lib/libprotobuf-lite.a  
2023-01-11T20:44:49.5736405Z   inflating: build/lib/libtensorpipe_uv.a  
2023-01-11T20:44:49.5833058Z   inflating: build/lib/libgloo.a     
2023-01-11T20:44:49.5892781Z   inflating: build/lib/libasmjit.a   
2023-01-11T20:44:49.6307659Z   inflating: build/lib/libprotobuf.a  
2023-01-11T20:44:49.6332468Z   inflating: build/lib/libfmt.a      
2023-01-11T20:44:49.6332950Z   inflating: build/lib/libfoxi_loader.a  
2023-01-11T20:44:49.6397332Z   inflating: build/lib/libc10.so     
2023-01-11T20:44:49.6398463Z   inflating: build/lib/libtorch_global_deps.so  
2023-01-11T20:44:49.6405870Z   inflating: build/lib/libcpuinfo.a  
2023-01-11T20:44:49.6412965Z   inflating: build/lib/libcpuinfo_internals.a  
2023-01-11T20:44:49.6425263Z   inflating: build/lib/libqnnpack.a  
2023-01-11T20:44:49.6443087Z   inflating: build/lib/libpytorch_qnnpack.a  
2023-01-11T20:44:49.6888033Z   inflating: build/lib/libprotoc.a   
2023-01-11T20:44:49.6889531Z   inflating: build/lib/libnnpack_reference_layers.a  
2023-01-11T20:44:49.6890229Z   inflating: build/lib/libgtest_main.a  
2023-01-11T20:44:49.6903945Z   inflating: build/lib/libgmock.a    
2023-01-11T20:44:49.6904356Z   inflating: build/lib/libbenchmark_main.a  
2023-01-11T20:44:50.4474702Z   inflating: build/lib/libdnnl.a     
2023-01-11T20:44:50.4491311Z   inflating: build/lib/libnnpack.a   
2023-01-11T20:44:50.4998402Z   inflating: build/lib/libtensorpipe.a  
2023-01-11T20:44:50.4998823Z   inflating: build/lib/libgmock_main.a  
2023-01-11T20:44:50.6185552Z   inflating: build/lib/libfbgemm.a   
2023-01-11T20:44:50.7086073Z   inflating: build/lib/libdnnl_graph.a  
2023-01-11T20:44:50.7319623Z   inflating: build/lib/libkineto.a   
2023-01-11T20:44:50.7354908Z   inflating: build/lib/libcaffe2_protos.a  
2023-01-11T20:44:50.7393516Z   inflating: build/lib/libonnx_proto.a  
2023-01-11T20:44:50.7505202Z   inflating: build/lib/libXNNPACK.a  
2023-01-11T20:44:50.8032808Z   inflating: build/lib/libonnx.a     
2023-01-11T20:44:52.6560582Z   inflating: build/lib/libtorch_cpu.so  
2023-01-11T20:44:52.6561281Z   inflating: build/lib/libtorch.so   
2023-01-11T20:44:52.6569422Z   inflating: build/lib/libunbox_lib.a  
2023-01-11T20:44:52.6573497Z   inflating: build/lib/libshm.so     
2023-01-11T20:44:52.6592857Z   inflating: build/lib/libjitbackend_test.so  
2023-01-11T20:44:52.6641002Z   inflating: build/lib/libtorchbind_test.so  
2023-01-11T20:44:52.6665392Z   inflating: build/lib/libbackend_with_compiler.so  
2023-01-11T20:44:52.8046110Z   inflating: build/lib/libtorch_python.so  
2023-01-11T20:44:52.8077474Z   inflating: build/lib/libnnapi_backend.so  
2023-01-11T20:44:52.8077899Z    creating: build/bin/
2023-01-11T20:44:52.8078267Z    creating: build/bin/CMakeFiles/
2023-01-11T20:44:52.8078785Z   inflating: build/bin/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T20:44:52.8079347Z  extracting: build/bin/CMakeFiles/progress.marks  
2023-01-11T20:44:52.8081960Z   inflating: build/bin/thread_init_test  
2023-01-11T20:44:52.8084274Z   inflating: build/bin/Makefile      
2023-01-11T20:44:52.8152041Z   inflating: build/bin/tensor_iterator_test  
2023-01-11T20:44:52.8152501Z   inflating: build/bin/cmake_install.cmake  
2023-01-11T20:44:52.8200789Z   inflating: build/bin/type_ptr_test  
2023-01-11T20:44:52.8201223Z   inflating: build/bin/CTestTestfile.cmake  
2023-01-11T20:44:52.8247004Z   inflating: build/bin/c10_DeviceGuard_test  
2023-01-11T20:44:52.8290224Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2023-01-11T20:44:52.8334655Z   inflating: build/bin/c10_Device_test  
2023-01-11T20:44:52.8376969Z   inflating: build/bin/c10_StreamGuard_test  
2023-01-11T20:44:52.8428197Z   inflating: build/bin/c10_DispatchKeySet_test  
2023-01-11T20:44:52.8472054Z   inflating: build/bin/c10_SymInt_test  
2023-01-11T20:44:52.8520908Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2023-01-11T20:44:52.8570029Z   inflating: build/bin/c10_InlineStreamGuard_test  
2023-01-11T20:44:52.8620161Z   inflating: build/bin/c10_SizesAndStrides_test  
2023-01-11T20:44:52.8662667Z   inflating: build/bin/c10_Array_test  
2023-01-11T20:44:52.8709631Z   inflating: build/bin/c10_Bitset_test  
2023-01-11T20:44:52.8754513Z   inflating: build/bin/c10_C++17_test  
2023-01-11T20:44:52.8796939Z   inflating: build/bin/c10_ConstexprCrc_test  
2023-01-11T20:44:52.8839946Z   inflating: build/bin/c10_DeadlockDetection_test  
2023-01-11T20:44:52.8884023Z   inflating: build/bin/c10_Half_test  
2023-01-11T20:44:52.8933827Z   inflating: build/bin/c10_LeftRight_test  
2023-01-11T20:44:52.8988367Z   inflating: build/bin/c10_Metaprogramming_test  
2023-01-11T20:44:52.9113733Z   inflating: build/bin/c10_SmallVectorTest  
2023-01-11T20:44:52.9158468Z   inflating: build/bin/c10_Synchronized_test  
2023-01-11T20:44:52.9208521Z   inflating: build/bin/c10_ThreadLocal_test  
2023-01-11T20:44:52.9255225Z   inflating: build/bin/c10_TypeIndex_test  
2023-01-11T20:44:52.9299863Z   inflating: build/bin/c10_TypeList_test  
2023-01-11T20:44:52.9342351Z   inflating: build/bin/c10_TypeTraits_test  
2023-01-11T20:44:52.9388389Z   inflating: build/bin/c10_accumulate_test  
2023-01-11T20:44:52.9437044Z   inflating: build/bin/c10_complex_math_test  
2023-01-11T20:44:52.9485460Z   inflating: build/bin/c10_complex_test  
2023-01-11T20:44:52.9534809Z   inflating: build/bin/c10_bfloat16_test  
2023-01-11T20:44:52.9629102Z   inflating: build/bin/c10_either_test  
2023-01-11T20:44:52.9673163Z   inflating: build/bin/c10_flags_test  
2023-01-11T20:44:52.9719613Z   inflating: build/bin/c10_exception_test  
2023-01-11T20:44:52.9863196Z   inflating: build/bin/c10_intrusive_ptr_test  
2023-01-11T20:44:52.9907925Z   inflating: build/bin/c10_irange_test  
2023-01-11T20:44:52.9957928Z   inflating: build/bin/c10_logging_test  
2023-01-11T20:44:53.0022268Z   inflating: build/bin/c10_optional_test  
2023-01-11T20:44:53.0076591Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2023-01-11T20:44:53.0125225Z   inflating: build/bin/c10_registry_test  
2023-01-11T20:44:53.0176202Z   inflating: build/bin/c10_string_view_test  
2023-01-11T20:44:53.0221978Z   inflating: build/bin/c10_tempfile_test  
2023-01-11T20:44:53.0271083Z   inflating: build/bin/c10_typeid_test  
2023-01-11T20:44:53.0319981Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2023-01-11T20:44:53.0732544Z   inflating: build/bin/protoc-3.13.0.0  
2023-01-11T20:44:53.1144504Z   inflating: build/bin/protoc        
2023-01-11T20:44:53.1397037Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2023-01-11T20:44:53.1677838Z   inflating: build/bin/vec_test_all_types_AVX2  
2023-01-11T20:44:53.1740585Z   inflating: build/bin/TCPStoreTest  
2023-01-11T20:44:53.1791724Z   inflating: build/bin/FileStoreTest  
2023-01-11T20:44:53.1839130Z   inflating: build/bin/HashStoreTest  
2023-01-11T20:44:53.1895977Z   inflating: build/bin/ProcessGroupGlooTest  
2023-01-11T20:44:53.1898577Z   inflating: build/bin/example_allreduce  
2023-01-11T20:44:53.1945869Z   inflating: build/bin/test_dist_autograd  
2023-01-11T20:44:53.1948285Z   inflating: build/bin/parallel_benchmark  
2023-01-11T20:44:53.2007793Z   inflating: build/bin/test_mobile_nnc  
2023-01-11T20:44:53.2068849Z   inflating: build/bin/test_cpp_rpc  
2023-01-11T20:44:53.2077756Z   inflating: build/bin/aot_model_compiler_test  
2023-01-11T20:44:53.2135760Z   inflating: build/bin/static_runtime_bench  
2023-01-11T20:44:53.2199173Z   inflating: build/bin/Dict_test     
2023-01-11T20:44:53.2245862Z   inflating: build/bin/Dimname_test  
2023-01-11T20:44:53.2301525Z   inflating: build/bin/MaybeOwned_test  
2023-01-11T20:44:53.2351811Z   inflating: build/bin/NamedTensor_test  
2023-01-11T20:44:53.2654650Z   inflating: build/bin/test_lazy     
2023-01-11T20:44:53.2706371Z   inflating: build/bin/apply_utils_test  
2023-01-11T20:44:53.2922071Z   inflating: build/bin/static_runtime_test  
2023-01-11T20:44:53.2973195Z   inflating: build/bin/atest         
2023-01-11T20:44:53.3025955Z   inflating: build/bin/basic         
2023-01-11T20:44:53.3073219Z   inflating: build/bin/broadcast_test  
2023-01-11T20:44:53.3123472Z   inflating: build/bin/cpu_generator_test  
2023-01-11T20:44:53.3169373Z   inflating: build/bin/cpu_profiling_allocator_test  
2023-01-11T20:44:53.3212898Z   inflating: build/bin/dispatch_key_set_test  
2023-01-11T20:44:53.3288867Z   inflating: build/bin/cpu_rng_test  
2023-01-11T20:44:53.3332254Z   inflating: build/bin/dlconvertor_test  
2023-01-11T20:44:53.3383146Z   inflating: build/bin/extension_backend_test  
2023-01-11T20:44:53.3431891Z   inflating: build/bin/half_test     
2023-01-11T20:44:53.3514030Z   inflating: build/bin/ivalue_test   
2023-01-11T20:44:53.3557067Z   inflating: build/bin/lazy_tensor_test  
2023-01-11T20:44:53.3604454Z   inflating: build/bin/math_kernel_test  
2023-01-11T20:44:53.3651671Z   inflating: build/bin/memory_format_test  
2023-01-11T20:44:53.3698805Z   inflating: build/bin/memory_overlapping_test  
2023-01-11T20:44:53.3747594Z   inflating: build/bin/native_test   
2023-01-11T20:44:53.3793833Z   inflating: build/bin/mobile_memory_cleanup  
2023-01-11T20:44:53.3838130Z   inflating: build/bin/operator_name_test  
2023-01-11T20:44:53.3881997Z   inflating: build/bin/operators_test  
2023-01-11T20:44:53.3928437Z   inflating: build/bin/packedtensoraccessor_test  
2023-01-11T20:44:53.3978829Z   inflating: build/bin/quantized_test  
2023-01-11T20:44:53.4993043Z   inflating: build/bin/test_api      
2023-01-11T20:44:53.5050824Z   inflating: build/bin/pow_test      
2023-01-11T20:44:53.5094072Z   inflating: build/bin/reduce_ops_test  
2023-01-11T20:44:53.5139207Z   inflating: build/bin/reportMemoryUsage_test  
2023-01-11T20:44:53.5189016Z   inflating: build/bin/scalar_tensor_test  
2023-01-11T20:44:53.5239120Z   inflating: build/bin/scalar_test   
2023-01-11T20:44:53.5285486Z   inflating: build/bin/stride_properties_test  
2023-01-11T20:44:53.5334998Z   inflating: build/bin/test_parallel  
2023-01-11T20:44:53.5378745Z   inflating: build/bin/variant_test  
2023-01-11T20:44:53.5432642Z   inflating: build/bin/type_test     
2023-01-11T20:44:53.5479187Z   inflating: build/bin/undefined_tensor_test  
2023-01-11T20:44:53.5480836Z   inflating: build/bin/verify_api_visibility  
2023-01-11T20:44:53.5541585Z   inflating: build/bin/legacy_vmap_test  
2023-01-11T20:44:53.5586850Z   inflating: build/bin/weakref_test  
2023-01-11T20:44:53.5632121Z   inflating: build/bin/wrapdim_test  
2023-01-11T20:44:53.5684936Z   inflating: build/bin/IListRef_test  
2023-01-11T20:44:53.5727839Z   inflating: build/bin/xla_tensor_test  
2023-01-11T20:44:53.5821927Z   inflating: build/bin/List_test     
2023-01-11T20:44:53.5927405Z   inflating: build/bin/kernel_function_legacy_test  
2023-01-11T20:44:53.6010821Z   inflating: build/bin/kernel_function_test  
2023-01-11T20:44:53.6067654Z   inflating: build/bin/KernelFunction_test  
2023-01-11T20:44:53.6178556Z   inflating: build/bin/kernel_lambda_legacy_test  
2023-01-11T20:44:53.6268093Z   inflating: build/bin/kernel_lambda_test  
2023-01-11T20:44:53.6321479Z   inflating: build/bin/kernel_stackbased_test  
2023-01-11T20:44:53.6366466Z   inflating: build/bin/CppSignature_test  
2023-01-11T20:44:53.6449656Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2023-01-11T20:44:53.6492001Z   inflating: build/bin/op_allowlist_test  
2023-01-11T20:44:53.6539693Z   inflating: build/bin/inline_container_test  
2023-01-11T20:44:53.6786043Z   inflating: build/bin/op_registration_test  
2023-01-11T20:44:53.6835366Z   inflating: build/bin/backend_fallback_test  
2023-01-11T20:44:53.6885744Z   inflating: build/bin/test_edge_op_registration  
2023-01-11T20:44:53.6891137Z   inflating: build/bin/torch_shm_manager  
2023-01-11T20:44:53.6905747Z   inflating: build/bin/tutorial_tensorexpr  
2023-01-11T20:44:53.7588358Z   inflating: build/bin/test_tensorexpr  
2023-01-11T20:44:53.8082928Z   inflating: build/bin/test_jit      
2023-01-11T20:44:53.8084446Z   inflating: .pytorch-test-times.json  
2023-01-11T20:44:53.8108543Z ##[group]Run df -H
2023-01-11T20:44:53.8108721Z df -H
2023-01-11T20:44:53.8119931Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:44:53.8120148Z env:
2023-01-11T20:44:53.8120328Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:53.8120513Z ##[endgroup]
2023-01-11T20:44:53.8153817Z Filesystem      Size  Used Avail Use% Mounted on
2023-01-11T20:44:53.8154122Z devtmpfs        8.2G     0  8.2G   0% /dev
2023-01-11T20:44:53.8154340Z tmpfs           8.2G  9.3M  8.2G   1% /dev/shm
2023-01-11T20:44:53.8154535Z tmpfs           8.2G  431k  8.2G   1% /run
2023-01-11T20:44:53.8154750Z tmpfs           8.2G     0  8.2G   0% /sys/fs/cgroup
2023-01-11T20:44:53.8154956Z /dev/nvme0n1p1  162G   35G  127G  22% /
2023-01-11T20:44:53.8214545Z ##[group]Run .github/scripts/parse_ref.py
2023-01-11T20:44:53.8214787Z .github/scripts/parse_ref.py
2023-01-11T20:44:53.8225576Z shell: /usr/bin/bash -e {0}
2023-01-11T20:44:53.8225759Z env:
2023-01-11T20:44:53.8225922Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:53.8226109Z ##[endgroup]
2023-01-11T20:44:53.8448612Z ##[group]Run set -x
2023-01-11T20:44:53.8448883Z set -x
2023-01-11T20:44:53.8449053Z 
2023-01-11T20:44:53.8449235Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2023-01-11T20:44:53.8449490Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2023-01-11T20:44:53.8449744Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2023-01-11T20:44:53.8449978Z   TEST_COMMAND=.jenkins/onnx/test.sh
2023-01-11T20:44:53.8450165Z else
2023-01-11T20:44:53.8450368Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2023-01-11T20:44:53.8450566Z fi
2023-01-11T20:44:53.8450711Z 
2023-01-11T20:44:53.8450945Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2023-01-11T20:44:53.8451190Z 
2023-01-11T20:44:53.8451435Z # sanitize the input commit message and PR body here:
2023-01-11T20:44:53.8451648Z #
2023-01-11T20:44:53.8451927Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2023-01-11T20:44:53.8452292Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2023-01-11T20:44:53.8452603Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2023-01-11T20:44:53.8452831Z PR_BODY="${PR_BODY//[$'\n\r']}"
2023-01-11T20:44:53.8453014Z 
2023-01-11T20:44:53.8453262Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2023-01-11T20:44:53.8453540Z # wreak havoc internally
2023-01-11T20:44:53.8453821Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2023-01-11T20:44:53.8454049Z export PR_BODY="${PR_BODY//[\'\"]}"
2023-01-11T20:44:53.8454238Z 
2023-01-11T20:44:53.8454463Z # detached container should get cleaned up by teardown_ec2_linux
2023-01-11T20:44:53.8454743Z # TODO: Stop building test binaries as part of the build phase
2023-01-11T20:44:53.8455020Z # Used for GPU_FLAG since that doesn't play nice
2023-01-11T20:44:53.8455256Z # shellcheck disable=SC2086,SC2090
2023-01-11T20:44:53.8455477Z container_name=$(docker run \
2023-01-11T20:44:53.8455666Z   ${GPU_FLAG:-} \
2023-01-11T20:44:53.8455863Z   -e BUILD_ENVIRONMENT \
2023-01-11T20:44:53.8456058Z   -e PR_NUMBER \
2023-01-11T20:44:53.8456237Z   -e GITHUB_ACTIONS \
2023-01-11T20:44:53.8456422Z   -e BASE_SHA \
2023-01-11T20:44:53.8456602Z   -e BRANCH \
2023-01-11T20:44:53.8456763Z   -e SHA1 \
2023-01-11T20:44:53.8456948Z   -e AWS_DEFAULT_REGION \
2023-01-11T20:44:53.8457143Z   -e IN_WHEEL_TEST \
2023-01-11T20:44:53.8457321Z   -e SHARD_NUMBER \
2023-01-11T20:44:53.8457508Z   -e TEST_CONFIG \
2023-01-11T20:44:53.8457726Z   -e NUM_TEST_SHARDS \
2023-01-11T20:44:53.8457916Z   -e PR_BODY \
2023-01-11T20:44:53.8458094Z   -e COMMIT_MESSAGES \
2023-01-11T20:44:53.8458298Z   -e CONTINUE_THROUGH_ERROR \
2023-01-11T20:44:53.8458513Z   -e PYTORCH_RETRY_TEST_CASES \
2023-01-11T20:44:53.8458728Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2023-01-11T20:44:53.8458935Z   -e PR_LABELS \
2023-01-11T20:44:53.8459145Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2023-01-11T20:44:53.8459343Z   -e SCCACHE_BUCKET \
2023-01-11T20:44:53.8459541Z   -e SCCACHE_S3_KEY_PREFIX \
2023-01-11T20:44:53.8459734Z   -e XLA_CUDA \
2023-01-11T20:44:53.8459926Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2023-01-11T20:44:53.8460159Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2023-01-11T20:44:53.8460396Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2023-01-11T20:44:53.8460785Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2023-01-11T20:44:53.8461007Z   --ulimit stack=10485760:83886080 \
2023-01-11T20:44:53.8461273Z   --security-opt seccomp=unconfined \
2023-01-11T20:44:53.8461498Z   --cap-add=SYS_PTRACE \
2023-01-11T20:44:53.8461872Z   --ipc=host \
2023-01-11T20:44:53.8462075Z   --shm-size="${SHM_SIZE}" \
2023-01-11T20:44:53.8462262Z   --tty \
2023-01-11T20:44:53.8462421Z   --detach \
2023-01-11T20:44:53.8462616Z   --name="${container_name}" \
2023-01-11T20:44:53.8462814Z   --user jenkins \
2023-01-11T20:44:53.8463034Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2023-01-11T20:44:53.8463283Z   -w /var/lib/jenkins/workspace \
2023-01-11T20:44:53.8463489Z   "${DOCKER_IMAGE}"
2023-01-11T20:44:53.8463663Z )
2023-01-11T20:44:53.8463869Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2023-01-11T20:44:53.8464207Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2023-01-11T20:44:53.8475345Z shell: /usr/bin/bash -e {0}
2023-01-11T20:44:53.8475520Z env:
2023-01-11T20:44:53.8475698Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:53.8475929Z   BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7
2023-01-11T20:44:53.8476131Z   PR_NUMBER: 91627
2023-01-11T20:44:53.8476308Z   BRANCH: pull/91627
2023-01-11T20:44:53.8476522Z   SHA1: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T20:44:53.8476756Z   BASE_SHA: db2a237763eb8693a20788be94f8c192e762baa8
2023-01-11T20:44:53.8476977Z   PYTORCH_RETRY_TEST_CASES: 1
2023-01-11T20:44:53.8477186Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2023-01-11T20:44:53.8477375Z   TEST_CONFIG: distributed
2023-01-11T20:44:53.8477557Z   SHARD_NUMBER: 1
2023-01-11T20:44:53.8477732Z   NUM_TEST_SHARDS: 2
2023-01-11T20:44:53.8477935Z   PR_BODY: Fixes #91003 


cc @ezyang @gchanan
2023-01-11T20:44:53.8478140Z   CONTINUE_THROUGH_ERROR: False
2023-01-11T20:44:53.8478384Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2023-01-11T20:44:53.8478624Z   SCCACHE_S3_KEY_PREFIX: pull
2023-01-11T20:44:53.8478799Z   SHM_SIZE: 1g
2023-01-11T20:44:53.8479139Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:44:53.8479466Z   XLA_CUDA: 
2023-01-11T20:44:53.8479711Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2023-01-11T20:44:53.8479992Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2023-01-11T20:44:53.8480209Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2023-01-11T20:44:53.8480390Z ##[endgroup]
2023-01-11T20:44:53.8504223Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2023-01-11T20:44:53.8504655Z + [[ linux-focal-py3.7-gcc7 == *onnx* ]]
2023-01-11T20:44:53.8504892Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2023-01-11T20:44:53.8507218Z ++ git cherry -v origin/master
2023-01-11T20:44:53.8943352Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into '\''input'\''
2023-01-11T20:44:53.8943772Z + 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch
2023-01-11T20:44:53.8944214Z + 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e '\''other'\'' instead of '\''output'\'' in documentation'
2023-01-11T20:44:53.8944998Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into '\''input'\''+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e '\''other'\'' instead of '\''output'\'' in documentation'
2023-01-11T20:44:53.8945522Z + PR_BODY='Fixes #91003 cc @ezyang @gchanan'
2023-01-11T20:44:53.8946198Z + export 'COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation'
2023-01-11T20:44:53.8947363Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation'
2023-01-11T20:44:53.8947870Z + export 'PR_BODY=Fixes #91003 cc @ezyang @gchanan'
2023-01-11T20:44:53.8948149Z + PR_BODY='Fixes #91003 cc @ezyang @gchanan'
2023-01-11T20:44:53.8953876Z +++ nproc --ignore=2
2023-01-11T20:44:53.8965073Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e CONTINUE_THROUGH_ERROR -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3896099317 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:45:05.0450243Z + container_name=ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T20:45:05.0450648Z + echo DOCKER_CONTAINER_ID=ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T20:45:05.0454527Z ++ echo dist/torch-2.0.0a0+git8419ddd-cp37-cp37m-linux_x86_64.whl
2023-01-11T20:45:05.0456848Z + docker exec -t ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747 sh -c 'pip install dist/torch-2.0.0a0+git8419ddd-cp37-cp37m-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh'
2023-01-11T20:45:05.4694077Z Processing ./dist/torch-2.0.0a0+git8419ddd-cp37-cp37m-linux_x86_64.whl
2023-01-11T20:45:05.8320439Z Requirement already satisfied: networkx in /opt/conda/lib/python3.7/site-packages (from torch==2.0.0a0+git8419ddd) (2.6.3)
2023-01-11T20:45:05.8323982Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==2.0.0a0+git8419ddd) (4.4.0)
2023-01-11T20:45:05.8327648Z Requirement already satisfied: sympy in /opt/conda/lib/python3.7/site-packages (from torch==2.0.0a0+git8419ddd) (1.10.1)
2023-01-11T20:45:05.8342008Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.7/site-packages (from torch==2.0.0a0+git8419ddd) (3.3.0)
2023-01-11T20:45:05.8407213Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.7/site-packages (from opt-einsum>=3.3->torch==2.0.0a0+git8419ddd) (1.18.5)
2023-01-11T20:45:05.8581040Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy->torch==2.0.0a0+git8419ddd) (1.2.1)
2023-01-11T20:45:06.5984419Z Installing collected packages: torch
2023-01-11T20:45:12.2523682Z Successfully installed torch-2.0.0a0+git8419ddd
2023-01-11T20:45:12.3951453Z + echo 'Environment variables:'
2023-01-11T20:45:12.3951806Z Environment variables:
2023-01-11T20:45:12.3951994Z + env
2023-01-11T20:45:12.3957222Z INSTALLED_DB=yes
2023-01-11T20:45:12.3957959Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T20:45:12.3958430Z CONTINUE_THROUGH_ERROR=False
2023-01-11T20:45:12.3958890Z BUILD_ENVIRONMENT=linux-focal-py3.7-gcc7
2023-01-11T20:45:12.3959360Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-01-11T20:45:12.3959565Z HOSTNAME=ecc7409477c1
2023-01-11T20:45:12.3959983Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_f699c233-d5c9-40e6-ba32-5f69ab50a320
2023-01-11T20:45:12.3960261Z GITHUB_ACTION=__self
2023-01-11T20:45:12.3960466Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-01-11T20:45:12.3960884Z GITHUB_RUN_NUMBER=77928
2023-01-11T20:45:12.3961077Z TEST_CONFIG=distributed
2023-01-11T20:45:12.3961270Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-01-11T20:45:12.3961537Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-01-11T20:45:12.3961865Z GITHUB_TRIGGERING_ACTOR=albanD
2023-01-11T20:45:12.3962171Z GITHUB_REF_TYPE=branch
2023-01-11T20:45:12.3962370Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-01-11T20:45:12.3962598Z BASE_SHA=db2a237763eb8693a20788be94f8c192e762baa8
2023-01-11T20:45:12.3962792Z XLA_CUDA=
2023-01-11T20:45:12.3963614Z ***
2023-01-11T20:45:12.3965039Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-01-11T20:45:12.3965287Z GITHUB_REPOSITORY_ID=65600975
2023-01-11T20:45:12.3965489Z GITHUB_ACTIONS=true
2023-01-11T20:45:12.3965706Z SHA1=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T20:45:12.3965939Z GITHUB_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c
2023-01-11T20:45:12.3966233Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/91627/merge
2023-01-11T20:45:12.3966484Z UCC_HOME=/usr
2023-01-11T20:45:12.3966668Z GITHUB_REF=refs/pull/91627/merge
2023-01-11T20:45:12.3966856Z SHARD_NUMBER=1
2023-01-11T20:45:12.3967042Z GITHUB_REF_PROTECTED=false
2023-01-11T20:45:12.3967232Z HOME=/var/lib/jenkins
2023-01-11T20:45:12.3967444Z GITHUB_API_URL=https://api.github.com
2023-01-11T20:45:12.3967678Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-01-11T20:45:12.3967876Z INSTALLED_THRIFT=
2023-01-11T20:45:12.3968033Z UCX_COMMIT=
2023-01-11T20:45:12.3968215Z SCCACHE_S3_KEY_PREFIX=pull
2023-01-11T20:45:12.3968400Z NUM_TEST_SHARDS=2
2023-01-11T20:45:12.3968557Z UCX_HOME=/usr
2023-01-11T20:45:12.3969011Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_f699c233-d5c9-40e6-ba32-5f69ab50a320
2023-01-11T20:45:12.3969315Z PYTORCH_RETRY_TEST_CASES=1
2023-01-11T20:45:12.3969703Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_f699c233-d5c9-40e6-ba32-5f69ab50a320
2023-01-11T20:45:12.3970141Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2023-01-11T20:45:12.3970420Z GITHUB_EVENT_NAME=pull_request
2023-01-11T20:45:12.3970605Z GITHUB_RUN_ID=3896099317
2023-01-11T20:45:12.3971027Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_f699c233-d5c9-40e6-ba32-5f69ab50a320
2023-01-11T20:45:12.3971333Z GITHUB_ACTOR=LucaLumetti
2023-01-11T20:45:12.3971519Z PR_NUMBER=91627
2023-01-11T20:45:12.3971680Z DESIRED_CUDA=
2023-01-11T20:45:12.3971857Z GITHUB_RUN_ATTEMPT=2
2023-01-11T20:45:12.3972088Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-01-11T20:45:12.3972289Z TERM=xterm
2023-01-11T20:45:12.3972486Z INSTALLED_VISION=yes
2023-01-11T20:45:12.3972667Z BRANCH=pull/91627
2023-01-11T20:45:12.3972844Z OPENSSL_ROOT_DIR=/opt/openssl
2023-01-11T20:45:12.3973044Z CUDA_PATH=/usr/local/cuda
2023-01-11T20:45:12.3973415Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-01-11T20:45:12.3973702Z GITHUB_SERVER_URL=https://github.com
2023-01-11T20:45:12.3973907Z UCC_COMMIT=
2023-01-11T20:45:12.3974082Z INSTALLED_ANDROID=
2023-01-11T20:45:12.3974240Z SHLVL=1
2023-01-11T20:45:12.3974400Z MAX_JOBS=6
2023-01-11T20:45:12.3974576Z GITHUB_ACTOR_ID=7543386
2023-01-11T20:45:12.3975034Z COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation
2023-01-11T20:45:12.3975500Z GITHUB_WORKFLOW_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c
2023-01-11T20:45:12.3975728Z GITHUB_REF_NAME=91627/merge
2023-01-11T20:45:12.3976068Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-01-11T20:45:12.3976311Z GITHUB_JOB=test
2023-01-11T20:45:12.3976508Z GITHUB_REPOSITORY=pytorch/pytorch
2023-01-11T20:45:12.3976715Z GITHUB_RETENTION_DAYS=90
2023-01-11T20:45:12.3976896Z OPENSSL_DIR=/opt/openssl
2023-01-11T20:45:12.3977099Z GITHUB_ACTION_REPOSITORY=
2023-01-11T20:45:12.3977538Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T20:45:12.3977876Z GITHUB_BASE_REF=master
2023-01-11T20:45:12.3978040Z CI=true
2023-01-11T20:45:12.3978220Z GITHUB_REPOSITORY_OWNER=pytorch
2023-01-11T20:45:12.3978423Z INSTALLED_PROTOBUF=yes
2023-01-11T20:45:12.3978598Z GITHUB_HEAD_REF=master
2023-01-11T20:45:12.3978777Z GITHUB_ACTION_REF=
2023-01-11T20:45:12.3979057Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-01-11T20:45:12.3979277Z GITHUB_WORKFLOW=pull
2023-01-11T20:45:12.3979474Z DEBIAN_FRONTEND=noninteractive
2023-01-11T20:45:12.3979886Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_f699c233-d5c9-40e6-ba32-5f69ab50a320
2023-01-11T20:45:12.3980183Z PR_BODY=Fixes #91003 cc @ezyang @gchanan
2023-01-11T20:45:12.3980375Z _=/usr/bin/env
2023-01-11T20:45:12.3980659Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2023-01-11T20:45:12.4117678Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.7/site-packages/torch
2023-01-11T20:45:12.4118339Z + TORCH_BIN_DIR=/opt/conda/lib/python3.7/site-packages/torch/bin
2023-01-11T20:45:12.4118955Z + TORCH_LIB_DIR=/opt/conda/lib/python3.7/site-packages/torch/lib
2023-01-11T20:45:12.4119594Z + TORCH_TEST_DIR=/opt/conda/lib/python3.7/site-packages/torch/test
2023-01-11T20:45:12.4119992Z + BUILD_DIR=build
2023-01-11T20:45:12.4120347Z + BUILD_RENAMED_DIR=build_renamed
2023-01-11T20:45:12.4120707Z + BUILD_BIN_DIR=build/bin
2023-01-11T20:45:12.4121034Z + export VALGRIND=ON
2023-01-11T20:45:12.4121286Z + VALGRIND=ON
2023-01-11T20:45:12.4121479Z + export TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T20:45:12.4121681Z + TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T20:45:12.4121954Z + [[ linux-focal-py3.7-gcc7 == *clang9* ]]
2023-01-11T20:45:12.4122229Z + [[ linux-focal-py3.7-gcc7 != *bazel* ]]
2023-01-11T20:45:12.4122786Z ++ realpath build/custom_test_artifacts
2023-01-11T20:45:12.4129041Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2023-01-11T20:45:12.4132113Z ++ dirname .jenkins/pytorch/test.sh
2023-01-11T20:45:12.4137662Z + source .jenkins/pytorch/common.sh
2023-01-11T20:45:12.4140584Z +++ dirname .jenkins/pytorch/common.sh
2023-01-11T20:45:12.4145539Z ++ source .jenkins/pytorch/common_utils.sh
2023-01-11T20:45:12.4147437Z +++ declare -f -t trap_add
2023-01-11T20:45:12.4152289Z ++ set -ex
2023-01-11T20:45:12.4152840Z ++ [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2023-01-11T20:45:12.4153192Z ++ BUILD_TEST_LIBTORCH=0
2023-01-11T20:45:12.4153728Z + echo 'Environment variables'
2023-01-11T20:45:12.4154093Z Environment variables
2023-01-11T20:45:12.4154418Z + env
2023-01-11T20:45:12.4160919Z INSTALLED_DB=yes
2023-01-11T20:45:12.4161537Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T20:45:12.4162003Z CONTINUE_THROUGH_ERROR=False
2023-01-11T20:45:12.4162560Z BUILD_ENVIRONMENT=linux-focal-py3.7-gcc7
2023-01-11T20:45:12.4162962Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-01-11T20:45:12.4163303Z HOSTNAME=ecc7409477c1
2023-01-11T20:45:12.4164088Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_f699c233-d5c9-40e6-ba32-5f69ab50a320
2023-01-11T20:45:12.4164621Z GITHUB_ACTION=__self
2023-01-11T20:45:12.4164974Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-01-11T20:45:12.4165335Z GITHUB_RUN_NUMBER=77928
2023-01-11T20:45:12.4165674Z TEST_CONFIG=distributed
2023-01-11T20:45:12.4166027Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-01-11T20:45:12.4166492Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-01-11T20:45:12.4166920Z GITHUB_TRIGGERING_ACTOR=albanD
2023-01-11T20:45:12.4167256Z GITHUB_REF_TYPE=branch
2023-01-11T20:45:12.4167564Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-01-11T20:45:12.4167903Z BASE_SHA=db2a237763eb8693a20788be94f8c192e762baa8
2023-01-11T20:45:12.4168219Z XLA_CUDA=
2023-01-11T20:45:12.4168657Z ***
2023-01-11T20:45:12.4168869Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-01-11T20:45:12.4169078Z GITHUB_REPOSITORY_ID=65600975
2023-01-11T20:45:12.4169445Z GITHUB_ACTIONS=true
2023-01-11T20:45:12.4169664Z SHA1=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T20:45:12.4169917Z GITHUB_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c
2023-01-11T20:45:12.4170257Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/91627/merge
2023-01-11T20:45:12.4170508Z UCC_HOME=/usr
2023-01-11T20:45:12.4170693Z GITHUB_REF=refs/pull/91627/merge
2023-01-11T20:45:12.4170869Z SHARD_NUMBER=1
2023-01-11T20:45:12.4171056Z GITHUB_REF_PROTECTED=false
2023-01-11T20:45:12.4171251Z HOME=/var/lib/jenkins
2023-01-11T20:45:12.4171457Z GITHUB_API_URL=https://api.github.com
2023-01-11T20:45:12.4171694Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-01-11T20:45:12.4171894Z INSTALLED_THRIFT=
2023-01-11T20:45:12.4172052Z UCX_COMMIT=
2023-01-11T20:45:12.4172231Z SCCACHE_S3_KEY_PREFIX=pull
2023-01-11T20:45:12.4172424Z NUM_TEST_SHARDS=2
2023-01-11T20:45:12.4172581Z UCX_HOME=/usr
2023-01-11T20:45:12.4173000Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_f699c233-d5c9-40e6-ba32-5f69ab50a320
2023-01-11T20:45:12.4173304Z TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T20:45:12.4173503Z PYTORCH_RETRY_TEST_CASES=1
2023-01-11T20:45:12.4173889Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_f699c233-d5c9-40e6-ba32-5f69ab50a320
2023-01-11T20:45:12.4174327Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2023-01-11T20:45:12.4174599Z GITHUB_EVENT_NAME=pull_request
2023-01-11T20:45:12.4174785Z GITHUB_RUN_ID=3896099317
2023-01-11T20:45:12.4175203Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_f699c233-d5c9-40e6-ba32-5f69ab50a320
2023-01-11T20:45:12.4175505Z GITHUB_ACTOR=LucaLumetti
2023-01-11T20:45:12.4175675Z PR_NUMBER=91627
2023-01-11T20:45:12.4175848Z DESIRED_CUDA=
2023-01-11T20:45:12.4176025Z GITHUB_RUN_ATTEMPT=2
2023-01-11T20:45:12.4176187Z VALGRIND=ON
2023-01-11T20:45:12.4176407Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-01-11T20:45:12.4176627Z TERM=xterm
2023-01-11T20:45:12.4176798Z INSTALLED_VISION=yes
2023-01-11T20:45:12.4176970Z BRANCH=pull/91627
2023-01-11T20:45:12.4177159Z OPENSSL_ROOT_DIR=/opt/openssl
2023-01-11T20:45:12.4177360Z CUDA_PATH=/usr/local/cuda
2023-01-11T20:45:12.4177716Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-01-11T20:45:12.4178014Z GITHUB_SERVER_URL=https://github.com
2023-01-11T20:45:12.4178212Z UCC_COMMIT=
2023-01-11T20:45:12.4178371Z INSTALLED_ANDROID=
2023-01-11T20:45:12.4178539Z SHLVL=1
2023-01-11T20:45:12.4178693Z MAX_JOBS=6
2023-01-11T20:45:12.4178851Z GITHUB_ACTOR_ID=7543386
2023-01-11T20:45:12.4179299Z COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation
2023-01-11T20:45:12.4179781Z GITHUB_WORKFLOW_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c
2023-01-11T20:45:12.4179996Z GITHUB_REF_NAME=91627/merge
2023-01-11T20:45:12.4180331Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-01-11T20:45:12.4180592Z GITHUB_JOB=test
2023-01-11T20:45:12.4180791Z GITHUB_REPOSITORY=pytorch/pytorch
2023-01-11T20:45:12.4180986Z GITHUB_RETENTION_DAYS=90
2023-01-11T20:45:12.4181182Z OPENSSL_DIR=/opt/openssl
2023-01-11T20:45:12.4181380Z GITHUB_ACTION_REPOSITORY=
2023-01-11T20:45:12.4181903Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T20:45:12.4182203Z GITHUB_BASE_REF=master
2023-01-11T20:45:12.4182381Z CI=true
2023-01-11T20:45:12.4182550Z GITHUB_REPOSITORY_OWNER=pytorch
2023-01-11T20:45:12.4182754Z INSTALLED_PROTOBUF=yes
2023-01-11T20:45:12.4182947Z GITHUB_HEAD_REF=master
2023-01-11T20:45:12.4183116Z GITHUB_ACTION_REF=
2023-01-11T20:45:12.4183403Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-01-11T20:45:12.4183721Z GITHUB_WORKFLOW=pull
2023-01-11T20:45:12.4183907Z DEBIAN_FRONTEND=noninteractive
2023-01-11T20:45:12.4184368Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_f699c233-d5c9-40e6-ba32-5f69ab50a320
2023-01-11T20:45:12.4184685Z PR_BODY=Fixes #91003 cc @ezyang @gchanan
2023-01-11T20:45:12.4184867Z _=/usr/bin/env
2023-01-11T20:45:12.4185082Z + echo 'Testing pytorch'
2023-01-11T20:45:12.4185266Z Testing pytorch
2023-01-11T20:45:12.4185450Z + export LANG=C.UTF-8
2023-01-11T20:45:12.4185642Z + LANG=C.UTF-8
2023-01-11T20:45:12.4185835Z + PR_NUMBER=91627
2023-01-11T20:45:12.4186030Z + [[ distributed == \d\e\f\a\u\l\t ]]
2023-01-11T20:45:12.4186238Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2023-01-11T20:45:12.4186511Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2023-01-11T20:45:12.4186728Z + [[ distributed == \s\l\o\w ]]
2023-01-11T20:45:12.4186995Z + [[ linux-focal-py3.7-gcc7 == *slow-gradcheck* ]]
2023-01-11T20:45:12.4187279Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]]
2023-01-11T20:45:12.4187551Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2023-01-11T20:45:12.4187756Z + [[ distributed == *crossref* ]]
2023-01-11T20:45:12.4187963Z + [[ distributed == *dynamo* ]]
2023-01-11T20:45:12.4188172Z + [[ distributed == *inductor* ]]
2023-01-11T20:45:12.4188416Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2023-01-11T20:45:12.4188691Z + [[ linux-focal-py3.7-gcc7 != *-bazel-* ]]
2023-01-11T20:45:12.4188954Z + pip_install --user ninja==1.10.2
2023-01-11T20:45:12.4189230Z + pip install --progress-bar off --user ninja==1.10.2
2023-01-11T20:45:12.8170252Z Collecting ninja==1.10.2
2023-01-11T20:45:12.8350587Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2023-01-11T20:45:13.5300773Z Installing collected packages: ninja
2023-01-11T20:45:13.5388769Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2023-01-11T20:45:13.5389401Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2023-01-11T20:45:13.5456377Z Successfully installed ninja-1.10.2
2023-01-11T20:45:13.6219685Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T20:45:13.6220454Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T20:45:13.6221281Z + [[ linux-focal-py3.7-gcc7 == *asan* ]]
2023-01-11T20:45:13.6221571Z + [[ linux-focal-py3.7-gcc7 == *-tsan* ]]
2023-01-11T20:45:13.6222000Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2023-01-11T20:45:13.6222222Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2023-01-11T20:45:13.6228092Z + [[ linux-focal-py3.7-gcc7 == *tbb* ]]
2023-01-11T20:45:13.6239156Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]]
2023-01-11T20:45:13.6239660Z + [[ linux-focal-py3.7-gcc7 == *-bazel-* ]]
2023-01-11T20:45:13.6240126Z + [[ linux-focal-py3.7-gcc7 == *-tsan* ]]
2023-01-11T20:45:13.6241791Z + cd test
2023-01-11T20:45:13.6242305Z + python -c 'import torch; print(torch.__config__.show())'
2023-01-11T20:45:14.7365855Z PyTorch built with:
2023-01-11T20:45:14.7366430Z   - GCC 7.5
2023-01-11T20:45:14.7366798Z   - C++ Version: 201703
2023-01-11T20:45:14.7367222Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2023-01-11T20:45:14.7367654Z   - Intel(R) MKL-DNN v2.7.2 (Git Hash fbec3e25a559ee252022ae066817b204e106a6ba)
2023-01-11T20:45:14.7367955Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-01-11T20:45:14.7368219Z   - LAPACK is enabled (usually provided by MKL)
2023-01-11T20:45:14.7368458Z   - NNPACK is enabled
2023-01-11T20:45:14.7368684Z   - CPU capability usage: AVX2
2023-01-11T20:45:14.7371335Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2023-01-11T20:45:14.7373158Z 
2023-01-11T20:45:14.9577032Z + cd test
2023-01-11T20:45:14.9577513Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2023-01-11T20:45:16.0360149Z ATen/Parallel:
2023-01-11T20:45:16.0372434Z 	at::get_num_threads() : 4
2023-01-11T20:45:16.0372706Z 	at::get_num_interop_threads() : 4
2023-01-11T20:45:16.0372928Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-01-11T20:45:16.0373135Z 	omp_get_max_threads() : 4
2023-01-11T20:45:16.0373646Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2023-01-11T20:45:16.0373937Z 	mkl_get_max_threads() : 4
2023-01-11T20:45:16.0374268Z Intel(R) MKL-DNN v2.7.2 (Git Hash fbec3e25a559ee252022ae066817b204e106a6ba)
2023-01-11T20:45:16.0374528Z std::thread::hardware_concurrency() : 8
2023-01-11T20:45:16.0374744Z Environment variables:
2023-01-11T20:45:16.0374958Z 	OMP_NUM_THREADS : [not set]
2023-01-11T20:45:16.0375155Z 	MKL_NUM_THREADS : [not set]
2023-01-11T20:45:16.0375343Z ATen parallel backend: OpenMP
2023-01-11T20:45:16.0375471Z 
2023-01-11T20:45:16.2563561Z + [[ distributed == *backward* ]]
2023-01-11T20:45:16.2564072Z + [[ distributed == *xla* ]]
2023-01-11T20:45:16.2564376Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2023-01-11T20:45:16.2564792Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]]
2023-01-11T20:45:16.2565026Z + [[ distributed == distributed ]]
2023-01-11T20:45:16.2565211Z + install_filelock
2023-01-11T20:45:16.2565400Z + pip_install filelock
2023-01-11T20:45:16.2565660Z + pip install --progress-bar off filelock
2023-01-11T20:45:16.6370486Z Collecting filelock
2023-01-11T20:45:16.6533398Z   Downloading filelock-3.9.0-py3-none-any.whl (9.7 kB)
2023-01-11T20:45:17.3640623Z Installing collected packages: filelock
2023-01-11T20:45:17.3917210Z Successfully installed filelock-3.9.0
2023-01-11T20:45:17.4648579Z + install_triton
2023-01-11T20:45:17.4648922Z + local commit
2023-01-11T20:45:17.4649208Z + [[ distributed == *rocm* ]]
2023-01-11T20:45:17.4651166Z ++ get_pinned_commit triton
2023-01-11T20:45:17.4651556Z ++ cat .github/ci_commit_pins/triton.txt
2023-01-11T20:45:17.4663165Z + commit=0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T20:45:17.4663781Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T20:45:17.4664355Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T20:45:17.8063283Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T20:45:17.8070043Z   Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-92_cm2hd
2023-01-11T20:45:17.8102021Z   Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-92_cm2hd
2023-01-11T20:45:18.6267954Z   Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331'
2023-01-11T20:45:18.6301261Z   Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T20:45:18.9570665Z   Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T20:45:19.2911273Z   Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T20:45:19.2912807Z   Running command git submodule update --init --recursive -q
2023-01-11T20:45:19.7978766Z   Preparing metadata (setup.py) ... [?25l- done
2023-01-11T20:45:19.9597923Z [?25hCollecting cmake
2023-01-11T20:45:19.9788880Z   Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB)
2023-01-11T20:45:20.2453281Z Requirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from triton==2.0.0) (3.9.0)
2023-01-11T20:45:20.2456110Z Requirement already satisfied: torch in /opt/conda/lib/python3.7/site-packages (from triton==2.0.0) (2.0.0a0+git8419ddd)
2023-01-11T20:45:20.2665633Z Requirement already satisfied: sympy in /opt/conda/lib/python3.7/site-packages (from torch->triton==2.0.0) (1.10.1)
2023-01-11T20:45:20.2669627Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch->triton==2.0.0) (4.4.0)
2023-01-11T20:45:20.2673011Z Requirement already satisfied: networkx in /opt/conda/lib/python3.7/site-packages (from torch->triton==2.0.0) (2.6.3)
2023-01-11T20:45:20.2842642Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy->torch->triton==2.0.0) (1.2.1)
2023-01-11T20:45:20.2902664Z Building wheels for collected packages: triton
2023-01-11T20:46:04.8099549Z   Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | / - done
2023-01-11T20:46:04.8468641Z [?25h  Created wheel for triton: filename=triton-2.0.0-cp37-cp37m-linux_x86_64.whl size=15370740 sha256=c14c991f5455d15dc2614ed9777e9cff304679f3d00e3918c5e4adb39568ac38
2023-01-11T20:46:04.8470384Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/aa/c5/3a/263a5ba27ed44796dab5f19b43135afb68beabac5d137652f7
2023-01-11T20:46:04.8494060Z Successfully built triton
2023-01-11T20:46:05.5516786Z Installing collected packages: cmake, triton
2023-01-11T20:46:06.7239107Z Successfully installed cmake-3.25.0 triton-2.0.0
2023-01-11T20:46:06.8996089Z + pip_install --user jinja2
2023-01-11T20:46:06.8996601Z + pip install --progress-bar off --user jinja2
2023-01-11T20:46:07.3204840Z Collecting jinja2
2023-01-11T20:46:07.3373476Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2023-01-11T20:46:07.3495638Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.7/site-packages (from jinja2) (2.1.1)
2023-01-11T20:46:08.0591941Z Installing collected packages: jinja2
2023-01-11T20:46:08.1352471Z Successfully installed jinja2-3.1.2
2023-01-11T20:46:08.2140402Z + test_distributed
2023-01-11T20:46:08.2140868Z + echo 'Testing distributed python tests'
2023-01-11T20:46:08.2141107Z Testing distributed python tests
2023-01-11T20:46:08.2141446Z + python test/run_test.py --distributed-tests --shard 1 2 --verbose
2023-01-11T20:46:09.7162542Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2023-01-11T20:46:09.8578720Z Ignoring disabled issues:  ['91003']
2023-01-11T20:46:09.8944296Z Found test time stats from artifacts
2023-01-11T20:46:09.8958919Z Selected tests:
2023-01-11T20:46:09.8959260Z  distributed/algorithms/quantization/test_quantization
2023-01-11T20:46:09.8959711Z  distributed/test_distributed_spawn
2023-01-11T20:46:09.8960088Z  distributed/test_nccl
2023-01-11T20:46:09.8960367Z  distributed/pipeline/sync/test_transparency
2023-01-11T20:46:09.8960604Z  distributed/pipeline/sync/test_pipeline
2023-01-11T20:46:09.8960903Z  distributed/pipeline/sync/test_phony
2023-01-11T20:46:09.8962819Z  distributed/pipeline/sync/test_inplace
2023-01-11T20:46:09.8963641Z  distributed/pipeline/sync/test_deferred_batch_norm
2023-01-11T20:46:09.8964186Z  distributed/pipeline/sync/test_checkpoint
2023-01-11T20:46:09.8964529Z  distributed/pipeline/sync/test_balance
2023-01-11T20:46:09.8964853Z  distributed/pipeline/sync/skip/test_tracker
2023-01-11T20:46:09.8965110Z  distributed/pipeline/sync/skip/test_portal
2023-01-11T20:46:09.8965516Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2023-01-11T20:46:09.8965968Z  distributed/pipeline/sync/skip/test_api
2023-01-11T20:46:09.8966434Z  distributed/optim/test_apply_optimizer_in_backward
2023-01-11T20:46:09.8966884Z  distributed/elastic/timer/api_test
2023-01-11T20:46:09.8967335Z  distributed/checkpoint/test_2d_model_state_checkpoint
2023-01-11T20:46:09.8967686Z  distributed/_shard/test_replicated_tensor
2023-01-11T20:46:09.8967922Z  distributed/checkpoint/test_nested_dict
2023-01-11T20:46:09.8968158Z  distributed/checkpoint/test_dedup_tensors
2023-01-11T20:46:09.8968384Z  distributed/checkpoint/test_traverse
2023-01-11T20:46:09.8968607Z  distributed/fsdp/test_utils
2023-01-11T20:46:09.8968822Z  distributed/checkpoint/test_utils
2023-01-11T20:46:09.8969023Z  distributed/fsdp/test_fsdp_fx
2023-01-11T20:46:09.8969262Z  distributed/checkpoint/test_file_system_checkpoint
2023-01-11T20:46:09.8969504Z  distributed/fsdp/test_fsdp_overlap
2023-01-11T20:46:09.8982455Z  distributed/_shard/sharding_spec/test_sharding_spec
2023-01-11T20:46:09.8982966Z  distributed/checkpoint/test_fsdp_optim_state
2023-01-11T20:46:09.8983433Z  distributed/fsdp/test_fsdp_uneven
2023-01-11T20:46:09.8983704Z  distributed/fsdp/test_fsdp_multiple_wrapping
2023-01-11T20:46:09.8983923Z  distributed/fsdp/test_fsdp_input
2023-01-11T20:46:09.8984201Z  distributed/test_c10d_error_logger
2023-01-11T20:46:09.8984492Z  distributed/fsdp/test_fsdp_memory
2023-01-11T20:46:09.8984812Z  distributed/elastic/timer/local_timer_test
2023-01-11T20:46:09.8985250Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2023-01-11T20:46:09.8985643Z  distributed/fsdp/test_fsdp_apply
2023-01-11T20:46:09.8986005Z  distributed/_tensor/test_math_ops
2023-01-11T20:46:09.8986381Z  distributed/test_c10d_object_collectives
2023-01-11T20:46:09.8986762Z  distributed/_composable/test_replicate
2023-01-11T20:46:09.8987146Z  distributed/fsdp/test_fsdp_tp_integration
2023-01-11T20:46:09.8987542Z  distributed/_tensor/test_tp_sharding_ops
2023-01-11T20:46:09.8987962Z  distributed/tensor/parallel/test_parallelize_api
2023-01-11T20:46:09.8988356Z  distributed/test_c10d_common
2023-01-11T20:46:09.8988731Z  distributed/_composable/test_compose
2023-01-11T20:46:09.8989169Z  distributed/fsdp/test_fsdp_flatten_params
2023-01-11T20:46:09.8989455Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2023-01-11T20:46:09.8989691Z  distributed/fsdp/test_fsdp_exec_order
2023-01-11T20:46:09.8989908Z  distributed/test_c10d_spawn_gloo
2023-01-11T20:46:09.8990152Z  distributed/checkpoint/test_file_system_checkpoint_cpu
2023-01-11T20:46:09.8990376Z  distributed/test_pg_wrapper
2023-01-11T20:46:09.8990610Z  distributed/elastic/multiprocessing/api_test
2023-01-11T20:46:09.8990851Z  distributed/_tensor/test_common_rules
2023-01-11T20:46:09.8991059Z  distributed/fsdp/test_fsdp_comm_hooks
2023-01-11T20:46:09.8991280Z  distributed/_tensor/test_dtensor
2023-01-11T20:46:09.8991494Z  distributed/fsdp/test_fsdp_misc
2023-01-11T20:46:09.8991709Z  distributed/_tensor/test_device_mesh
2023-01-11T20:46:09.9008281Z  distributed/optim/test_zero_redundancy_optimizer
2023-01-11T20:46:09.9008593Z  distributed/test_c10d_pypg
2023-01-11T20:46:09.9008815Z  distributed/rpc/test_tensorpipe_agent
2023-01-11T20:46:09.9322060Z Prioritized test from test file changes.
2023-01-11T20:46:09.9322484Z reordering tests for PR:
2023-01-11T20:46:09.9323333Z prioritized: ['distributed/checkpoint/test_2d_model_state_checkpoint', 'distributed/_tensor/test_device_mesh']
2023-01-11T20:46:09.9331286Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/test_nccl', 'distributed/pipeline/sync/test_transparency', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_api', 'distributed/optim/test_apply_optimizer_in_backward', 'distributed/elastic/timer/api_test', 'distributed/_shard/test_replicated_tensor', 'distributed/checkpoint/test_nested_dict', 'distributed/checkpoint/test_dedup_tensors', 'distributed/checkpoint/test_traverse', 'distributed/fsdp/test_utils', 'distributed/checkpoint/test_utils', 'distributed/fsdp/test_fsdp_fx', 'distributed/checkpoint/test_file_system_checkpoint', 'distributed/fsdp/test_fsdp_overlap', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/checkpoint/test_fsdp_optim_state', 'distributed/fsdp/test_fsdp_uneven', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/fsdp/test_fsdp_input', 'distributed/test_c10d_error_logger', 'distributed/fsdp/test_fsdp_memory', 'distributed/elastic/timer/local_timer_test', 'distributed/_shard/sharded_tensor/test_sharded_tensor', 'distributed/fsdp/test_fsdp_apply', 'distributed/_tensor/test_math_ops', 'distributed/test_c10d_object_collectives', 'distributed/_composable/test_replicate', 'distributed/fsdp/test_fsdp_tp_integration', 'distributed/_tensor/test_tp_sharding_ops', 'distributed/tensor/parallel/test_parallelize_api', 'distributed/test_c10d_common', 'distributed/_composable/test_compose', 'distributed/fsdp/test_fsdp_flatten_params', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/fsdp/test_fsdp_exec_order', 'distributed/test_c10d_spawn_gloo', 'distributed/checkpoint/test_file_system_checkpoint_cpu', 'distributed/test_pg_wrapper', 'distributed/elastic/multiprocessing/api_test', 'distributed/_tensor/test_common_rules', 'distributed/fsdp/test_fsdp_comm_hooks', 'distributed/_tensor/test_dtensor', 'distributed/fsdp/test_fsdp_misc', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/test_c10d_pypg', 'distributed/rpc/test_tensorpipe_agent']
2023-01-11T20:46:09.9335946Z 
2023-01-11T20:46:09.9336717Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2023-01-11T20:46:09.9513955Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2023-01-11T20:46:09.9642930Z parallel (file granularity) tests:
2023-01-11T20:46:09.9643260Z  
2023-01-11T20:46:09.9643541Z serial (file granularity) tests:
2023-01-11T20:46:09.9644966Z  distributed/checkpoint/test_2d_model_state_checkpoint
2023-01-11T20:46:09.9645384Z  distributed/_tensor/test_device_mesh
2023-01-11T20:46:09.9645786Z  distributed/algorithms/quantization/test_quantization
2023-01-11T20:46:09.9646217Z  distributed/test_distributed_spawn
2023-01-11T20:46:09.9646559Z  distributed/test_nccl
2023-01-11T20:46:09.9646915Z  distributed/pipeline/sync/test_transparency
2023-01-11T20:46:09.9647315Z  distributed/pipeline/sync/test_pipeline
2023-01-11T20:46:09.9647703Z  distributed/pipeline/sync/test_phony
2023-01-11T20:46:09.9648107Z  distributed/pipeline/sync/test_inplace
2023-01-11T20:46:09.9648527Z  distributed/pipeline/sync/test_deferred_batch_norm
2023-01-11T20:46:09.9648958Z  distributed/pipeline/sync/test_checkpoint
2023-01-11T20:46:09.9649331Z  distributed/pipeline/sync/test_balance
2023-01-11T20:46:09.9649735Z  distributed/pipeline/sync/skip/test_tracker
2023-01-11T20:46:09.9650130Z  distributed/pipeline/sync/skip/test_portal
2023-01-11T20:46:09.9650552Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2023-01-11T20:46:09.9650967Z  distributed/pipeline/sync/skip/test_api
2023-01-11T20:46:09.9651357Z  distributed/optim/test_apply_optimizer_in_backward
2023-01-11T20:46:09.9651918Z  distributed/elastic/timer/api_test
2023-01-11T20:46:09.9652315Z  distributed/_shard/test_replicated_tensor
2023-01-11T20:46:09.9652694Z  distributed/checkpoint/test_nested_dict
2023-01-11T20:46:09.9653191Z  distributed/checkpoint/test_dedup_tensors
2023-01-11T20:46:09.9653610Z  distributed/checkpoint/test_traverse
2023-01-11T20:46:09.9653936Z  distributed/fsdp/test_utils
2023-01-11T20:46:09.9654305Z  distributed/checkpoint/test_utils
2023-01-11T20:46:09.9654654Z  distributed/fsdp/test_fsdp_fx
2023-01-11T20:46:09.9655030Z  distributed/checkpoint/test_file_system_checkpoint
2023-01-11T20:46:09.9655432Z  distributed/fsdp/test_fsdp_overlap
2023-01-11T20:46:09.9655835Z  distributed/_shard/sharding_spec/test_sharding_spec
2023-01-11T20:46:09.9656247Z  distributed/checkpoint/test_fsdp_optim_state
2023-01-11T20:46:09.9656639Z  distributed/fsdp/test_fsdp_uneven
2023-01-11T20:46:09.9657022Z  distributed/fsdp/test_fsdp_multiple_wrapping
2023-01-11T20:46:09.9657396Z  distributed/fsdp/test_fsdp_input
2023-01-11T20:46:09.9657783Z  distributed/test_c10d_error_logger
2023-01-11T20:46:09.9658173Z  distributed/fsdp/test_fsdp_memory
2023-01-11T20:46:09.9658563Z  distributed/elastic/timer/local_timer_test
2023-01-11T20:46:09.9658980Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2023-01-11T20:46:09.9659361Z  distributed/fsdp/test_fsdp_apply
2023-01-11T20:46:09.9659721Z  distributed/_tensor/test_math_ops
2023-01-11T20:46:09.9660058Z  distributed/test_c10d_object_collectives
2023-01-11T20:46:09.9660403Z  distributed/_composable/test_replicate
2023-01-11T20:46:09.9660789Z  distributed/fsdp/test_fsdp_tp_integration
2023-01-11T20:46:09.9661165Z  distributed/_tensor/test_tp_sharding_ops
2023-01-11T20:46:09.9661580Z  distributed/tensor/parallel/test_parallelize_api
2023-01-11T20:46:09.9662186Z  distributed/test_c10d_common
2023-01-11T20:46:09.9662536Z  distributed/_composable/test_compose
2023-01-11T20:46:09.9662913Z  distributed/fsdp/test_fsdp_flatten_params
2023-01-11T20:46:09.9663449Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2023-01-11T20:46:09.9663843Z  distributed/fsdp/test_fsdp_exec_order
2023-01-11T20:46:09.9664226Z  distributed/test_c10d_spawn_gloo
2023-01-11T20:46:09.9664656Z  distributed/checkpoint/test_file_system_checkpoint_cpu
2023-01-11T20:46:09.9665024Z  distributed/test_pg_wrapper
2023-01-11T20:46:09.9665472Z  distributed/elastic/multiprocessing/api_test
2023-01-11T20:46:09.9665867Z  distributed/_tensor/test_common_rules
2023-01-11T20:46:09.9666241Z  distributed/fsdp/test_fsdp_comm_hooks
2023-01-11T20:46:09.9666578Z  distributed/_tensor/test_dtensor
2023-01-11T20:46:09.9666932Z  distributed/fsdp/test_fsdp_misc
2023-01-11T20:46:09.9667350Z  distributed/optim/test_zero_redundancy_optimizer
2023-01-11T20:46:09.9667723Z  distributed/test_c10d_pypg
2023-01-11T20:46:09.9668079Z  distributed/rpc/test_tensorpipe_agent
2023-01-11T20:46:11.4874752Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2023-01-11T20:46:11.4875386Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2023-01-11T20:46:11.6096310Z Ignoring disabled issues:  ['91003']
2023-01-11T20:46:11.6292958Z Ignoring disabled issues:  ['91003']
2023-01-11T20:46:12.0048297Z Running distributed/checkpoint/test_2d_model_state_checkpoint ... [2023-01-11 20:46:12.004415]
2023-01-11T20:46:12.0051114Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_2d_model_state_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:46:12.004847]
2023-01-11T20:46:16.3668303Z 
2023-01-11T20:46:16.3668815Z Expand the folded group to see the log file of distributed/checkpoint/test_2d_model_state_checkpoint
2023-01-11T20:46:16.3673985Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_2d_model_state_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_2d_model_state_checkpoint_2o1ykg5c)
2023-01-11T20:46:16.3674756Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_2d_model_state_checkpoint
2023-01-11T20:46:16.3675294Z 
2023-01-11T20:46:16.3675410Z Running tests...
2023-01-11T20:46:16.3675840Z ----------------------------------------------------------------------
2023-01-11T20:46:16.3676616Z   test_2d_model_state_checkpoint (__main__.Test2dModelStateCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 701
2023-01-11T20:46:16.3677211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 702
2023-01-11T20:46:16.3677544Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 703
2023-01-11T20:46:16.3677852Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 704
2023-01-11T20:46:16.3678317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.3678667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.3679107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.3679443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.3679771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:46:16.3680231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.3680545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.3680976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.3681319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.3681641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:46:16.3682087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.3682418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.3682846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.3683189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.3683497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:46:16.3684056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.3684424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.3684842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.3685185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.3685511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:46:16.3685884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:46:16.3686245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:46:16.3686609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:46:16.3686973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:46:16.3687455Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.3687977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.3688490Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.3689060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.3689366Z skip: Need at least 4 CUDA devices (2.698s)
2023-01-11T20:46:16.3689540Z 
2023-01-11T20:46:16.3689743Z ----------------------------------------------------------------------
2023-01-11T20:46:16.3689987Z Ran 1 test in 2.698s
2023-01-11T20:46:16.3690104Z 
2023-01-11T20:46:16.3690178Z OK (skipped=1)
2023-01-11T20:46:16.3690271Z 
2023-01-11T20:46:16.3690356Z Generating XML reports...
2023-01-11T20:46:16.3690873Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_2d_model_state_checkpoint/TEST-Test2dModelStateCheckpoint-20230111204613.xml
2023-01-11T20:46:16.3691180Z 
2023-01-11T20:46:16.3691476Z ##[endgroup]
2023-01-11T20:46:16.3692036Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_2d_model_state_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_2d_model_state_checkpoint_2o1ykg5c)
2023-01-11T20:46:16.3692344Z 
2023-01-11T20:46:16.3692544Z Running distributed/_tensor/test_device_mesh ... [2023-01-11 20:46:16.367093]
2023-01-11T20:46:16.3693082Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_device_mesh.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:46:16.367448]
2023-01-11T20:47:14.1562701Z 
2023-01-11T20:47:14.1563180Z Expand the folded group to see the log file of distributed/_tensor/test_device_mesh
2023-01-11T20:47:14.1564036Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_device_mesh (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_device_mesh_ww9x5n7u)
2023-01-11T20:47:14.1567811Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_device_mesh
2023-01-11T20:47:14.1568181Z 
2023-01-11T20:47:14.1568294Z Running tests...
2023-01-11T20:47:14.1568813Z ----------------------------------------------------------------------
2023-01-11T20:47:14.1569442Z   test_all_gather_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 768
2023-01-11T20:47:14.1570103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 769
2023-01-11T20:47:14.1570651Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 770
2023-01-11T20:47:14.1571158Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 771
2023-01-11T20:47:14.1571632Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 772
2023-01-11T20:47:14.1572114Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 773
2023-01-11T20:47:14.1572473Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 774
2023-01-11T20:47:14.1572973Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 775
2023-01-11T20:47:14.1573746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1574273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1574888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1575372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1575828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.1576550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1577093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1577790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1578314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1578699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.1579605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1580173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1580853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1581356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1582119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.1582857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1583243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1583857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1584334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1584777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.1585504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1585964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1586626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1587206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1587657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.1588415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1588945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1589575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1590078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1590541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.1591108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1591630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1592419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1593034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1593600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.1594457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1594946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1595675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1596311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1596828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.1597403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.1597997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.1598567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.1599296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.1599790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.1600427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.1600914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.1601425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.1602233Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1603041Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1603849Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1604659Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1605415Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1606189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1606963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1607699Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1608126Z ok (3.106s)
2023-01-11T20:47:14.1608532Z   test_all_gather_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 880
2023-01-11T20:47:14.1609187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 881
2023-01-11T20:47:14.1609678Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 882
2023-01-11T20:47:14.1610182Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 883
2023-01-11T20:47:14.1610710Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 884
2023-01-11T20:47:14.1611239Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 885
2023-01-11T20:47:14.1611690Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 886
2023-01-11T20:47:14.1612201Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 887
2023-01-11T20:47:14.1612960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1613424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1614052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1614561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1615013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.1615668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1616110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1616640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1617175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1618761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.1619242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1619668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1620133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1620482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1620806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.1621262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1621575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1622155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1622499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1622814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.1623275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1623600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1624107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1624437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1624757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.1625216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1625543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1625957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1626308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1626633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.1627078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1627401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1627830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1628176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1628482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.1628938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1629266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1629682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1630023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1630342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.1630703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.1631057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.1631418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.1631777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.1632206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.1632608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.1632967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.1633466Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1633849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.1634336Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1634847Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1635360Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1635861Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1636249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T20:47:14.1636612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T20:47:14.1636969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:47:14.1637314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T20:47:14.1637800Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1638313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1638826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1639223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:47:14.1639585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:47:14.1639945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T20:47:14.1640303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:47:14.1640779Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1641283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1641675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:47:14.1642022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T20:47:14.1642508Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1643025Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1643540Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1643990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T20:47:14.1644477Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1644865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:47:14.1645265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T20:47:14.1645767Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1646280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1646668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:47:14.1647029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:47:14.1647373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T20:47:14.1647860Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1648374Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1648882Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1649379Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1649888Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1650280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:47:14.1650640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T20:47:14.1650986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:47:14.1651342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T20:47:14.1651828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1652321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1652829Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1653220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:47:14.1653579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T20:47:14.1653925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:47:14.1654281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T20:47:14.1654769Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1655277Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1655774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1656161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T20:47:14.1656637Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1657026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:47:14.1657497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1658061Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1658453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:47:14.1658802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T20:47:14.1659288Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1659797Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1660187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T20:47:14.1660537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:47:14.1660895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:47:14.1661386Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1661957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T20:47:14.1662438Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1662827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T20:47:14.1663187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2023-01-11T20:47:14.1663673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1664174Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1664566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2023-01-11T20:47:14.1665051Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1665564Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1666056Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1666563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1666949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2023-01-11T20:47:14.1667312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2023-01-11T20:47:14.1667664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T20:47:14.1668022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T20:47:14.1668372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T20:47:14.1668845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1669353Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1669740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T20:47:14.1670101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2023-01-11T20:47:14.1670640Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1671206Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1671718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1672103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2023-01-11T20:47:14.1672570Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1672953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2023-01-11T20:47:14.1673435Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1673828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2023-01-11T20:47:14.1674298Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1674682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2023-01-11T20:47:14.1675039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T20:47:14.1675384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2023-01-11T20:47:14.1675868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1676377Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1676767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2023-01-11T20:47:14.1677116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3
2023-01-11T20:47:14.1677597Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1678105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1678613Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1678984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2
2023-01-11T20:47:14.1679344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2023-01-11T20:47:14.1679830Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1680344Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1680845Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1681232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5
2023-01-11T20:47:14.1681591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7
2023-01-11T20:47:14.1681947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4
2023-01-11T20:47:14.1682290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6
2023-01-11T20:47:14.1682876Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1683431Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1684038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1684416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6
2023-01-11T20:47:14.1684776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4
2023-01-11T20:47:14.1685135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2023-01-11T20:47:14.1685605Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1686114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1686625Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1687134Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1687580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2023-01-11T20:47:14.1687939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3
2023-01-11T20:47:14.1688425Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1688814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2
2023-01-11T20:47:14.1689161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5
2023-01-11T20:47:14.1689522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7
2023-01-11T20:47:14.1690007Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1690405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7
2023-01-11T20:47:14.1690874Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1691422Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1691934Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1692441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1692819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6
2023-01-11T20:47:14.1693256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2
2023-01-11T20:47:14.1693621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4
2023-01-11T20:47:14.1694092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1694599Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1694986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3
2023-01-11T20:47:14.1695465Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1695837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5
2023-01-11T20:47:14.1696240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0
2023-01-11T20:47:14.1696629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1
2023-01-11T20:47:14.1697123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1697622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1698123Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1698513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0
2023-01-11T20:47:14.1698873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1
2023-01-11T20:47:14.1699227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5
2023-01-11T20:47:14.1699718Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1700108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7
2023-01-11T20:47:14.1700595Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1701092Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1701593Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1702197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6
2023-01-11T20:47:14.1702563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4
2023-01-11T20:47:14.1703044Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1703433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2
2023-01-11T20:47:14.1703794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3
2023-01-11T20:47:14.1704268Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1704777Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1705164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2
2023-01-11T20:47:14.1705531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3
2023-01-11T20:47:14.1706007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1706510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1706900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0
2023-01-11T20:47:14.1707386Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1707759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1
2023-01-11T20:47:14.1708243Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1708632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7
2023-01-11T20:47:14.1709205Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1709740Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1710129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6
2023-01-11T20:47:14.1710501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4
2023-01-11T20:47:14.1710861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5
2023-01-11T20:47:14.1711332Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1711842Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1712232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4
2023-01-11T20:47:14.1712584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5
2023-01-11T20:47:14.1713070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1713577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1713962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0
2023-01-11T20:47:14.1714313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1
2023-01-11T20:47:14.1714801Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1715310Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1715820Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1716316Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1716701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2
2023-01-11T20:47:14.1717060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3
2023-01-11T20:47:14.1717424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6
2023-01-11T20:47:14.1717768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7
2023-01-11T20:47:14.1718255Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1718765Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1719269Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1719757Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1720260Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1720768Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1721268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1721803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1722113Z ok (3.040s)
2023-01-11T20:47:14.1722445Z   test_all_gather_uneven (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1064
2023-01-11T20:47:14.1722830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1065
2023-01-11T20:47:14.1723155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1066
2023-01-11T20:47:14.1723470Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1067
2023-01-11T20:47:14.1723854Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1068
2023-01-11T20:47:14.1724161Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1069
2023-01-11T20:47:14.1724477Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1070
2023-01-11T20:47:14.1724805Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1071
2023-01-11T20:47:14.1725248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1725580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1726012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1726357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1726665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.1727124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1727448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1727871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1728205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1728528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.1728988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1729304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1729727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1730068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1730387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.1730834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1731162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1731593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1731921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1732242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.1732699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1733037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1733448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1733789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1734161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.1734647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1734961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1735383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1735725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1736032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.1736493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1736818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1737243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1737571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1737892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.1738347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1738658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1739080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1739421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1739741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.1740092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.1740456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.1740814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.1741174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.1741518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.1742018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.1742375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.1742715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.1743208Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1743732Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1744250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1744749Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1745266Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1745778Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1746286Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1746781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1747140Z ok (2.839s)
2023-01-11T20:47:14.1747503Z   test_all_reduce_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1176
2023-01-11T20:47:14.1747904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1177
2023-01-11T20:47:14.1748218Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1178
2023-01-11T20:47:14.1748536Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1179
2023-01-11T20:47:14.1748853Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1180
2023-01-11T20:47:14.1749160Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1181
2023-01-11T20:47:14.1749473Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1182
2023-01-11T20:47:14.1749789Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1183
2023-01-11T20:47:14.1750246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1750562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1750989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1751332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1751640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.1752099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1752425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1752851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1753181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1753501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.1753959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1754270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1754694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1755033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1755353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.1755794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1756123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1756547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1756892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1757199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.1757652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1757972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1758382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1758716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1759037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.1759536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1759846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1760317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1760656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1760962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.1761415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1761737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1762157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1762484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1762805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.1763259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1763583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1764074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1764414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1764735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.1765084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.1765450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.1765824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.1766179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.1766532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.1766884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.1767227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.1767588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.1768075Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1768595Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1769097Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1769614Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1770133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1770648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1771186Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1771704Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1771989Z ok (2.839s)
2023-01-11T20:47:14.1772364Z   test_all_reduce_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1288
2023-01-11T20:47:14.1772777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1289
2023-01-11T20:47:14.1773106Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1290
2023-01-11T20:47:14.1773426Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1291
2023-01-11T20:47:14.1773733Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1292
2023-01-11T20:47:14.1774059Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1293
2023-01-11T20:47:14.1774379Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1294
2023-01-11T20:47:14.1774699Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1295
2023-01-11T20:47:14.1775136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1775471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1775903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1776255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1776566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.1777032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1777361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1777782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1778133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1778456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.1778920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1779233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1779661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1780003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1780311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.1780769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1781093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1781518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1783181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1783510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.1783976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1784301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1784826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1785171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1785489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.1785936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1786364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1786832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1787180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1787492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.1787953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1788278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1788689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1789033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1789358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.1789815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1790127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1790552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1790906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1791224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.1791574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.1791940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.1792301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.1792649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.1793007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.1793360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.1793710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.1794059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.1794545Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1795063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1795577Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1796081Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1796587Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1797094Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1797598Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1798094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1798480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:47:14.1798884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T20:47:14.1799256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:47:14.1799613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T20:47:14.1799968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T20:47:14.1800327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:47:14.1800673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:47:14.1801028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T20:47:14.1801510Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1802045Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1802545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1803050Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1803555Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1804140Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1804517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:47:14.1805004Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1805392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T20:47:14.1805755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T20:47:14.1806100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T20:47:14.1806461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:47:14.1806943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1807319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:47:14.1807677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:47:14.1808157Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1808544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T20:47:14.1809013Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1809520Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1809907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:47:14.1810388Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1810763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T20:47:14.1811123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T20:47:14.1811523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:47:14.1812034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1812533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1813040Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1813423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:47:14.1813892Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1814274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T20:47:14.1814633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:47:14.1814989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T20:47:14.1815456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1815961Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1816345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:47:14.1816702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T20:47:14.1817177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1817692Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1818206Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1818708Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1819080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:47:14.1819559Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1819941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T20:47:14.1820421Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1820818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T20:47:14.1821176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:47:14.1821537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:47:14.1822042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T20:47:14.1822534Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1823045Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1823432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T20:47:14.1823775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2023-01-11T20:47:14.1824326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1824873Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1825264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T20:47:14.1825615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2023-01-11T20:47:14.1826100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1826608Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1827116Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1827609Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1827994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2023-01-11T20:47:14.1828350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2023-01-11T20:47:14.1828704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T20:47:14.1829043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T20:47:14.1829524Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1830035Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1830544Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1830927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2023-01-11T20:47:14.1831410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1831797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2023-01-11T20:47:14.1832146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2023-01-11T20:47:14.1832502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T20:47:14.1832993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1833500Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1833882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2023-01-11T20:47:14.1834370Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1834758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2023-01-11T20:47:14.1835242Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1835621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T20:47:14.1835979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2023-01-11T20:47:14.1836462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1837010Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1837433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2023-01-11T20:47:14.1837798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3
2023-01-11T20:47:14.1838283Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1838782Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1839296Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1839684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2
2023-01-11T20:47:14.1840166Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1840547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7
2023-01-11T20:47:14.1841034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1841427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2023-01-11T20:47:14.1841908Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1842289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5
2023-01-11T20:47:14.1842645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4
2023-01-11T20:47:14.1843002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6
2023-01-11T20:47:14.1843489Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1844074Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1844584Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1844977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4
2023-01-11T20:47:14.1845337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6
2023-01-11T20:47:14.1845811Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1846198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2
2023-01-11T20:47:14.1846682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1847075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2023-01-11T20:47:14.1847546Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1848045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1848554Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.1848923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2023-01-11T20:47:14.1849278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3
2023-01-11T20:47:14.1849694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5
2023-01-11T20:47:14.1850093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7
2023-01-11T20:47:14.1850563Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1851070Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1851462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7
2023-01-11T20:47:14.1851942Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1852320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5
2023-01-11T20:47:14.1852803Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1853310Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1853702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4
2023-01-11T20:47:14.1854050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2
2023-01-11T20:47:14.1854539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1854927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6
2023-01-11T20:47:14.1855406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1855902Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.1856297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3
2023-01-11T20:47:14.1856661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0
2023-01-11T20:47:14.1857012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1
2023-01-11T20:47:14.1857492Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1858005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1858508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1858888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5
2023-01-11T20:47:14.1859376Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1859880Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1860268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1
2023-01-11T20:47:14.1860615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0
2023-01-11T20:47:14.1861103Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1861490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4
2023-01-11T20:47:14.1861996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6
2023-01-11T20:47:14.1862541Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1863090Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.1863484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7
2023-01-11T20:47:14.1863848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2
2023-01-11T20:47:14.1864199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3
2023-01-11T20:47:14.1864689Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1865194Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1865688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1866079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2
2023-01-11T20:47:14.1866563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1866957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3
2023-01-11T20:47:14.1867432Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1867945Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1868328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0
2023-01-11T20:47:14.1868698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1
2023-01-11T20:47:14.1869050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6
2023-01-11T20:47:14.1869536Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1870040Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.1870431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7
2023-01-11T20:47:14.1870778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4
2023-01-11T20:47:14.1871140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5
2023-01-11T20:47:14.1871621Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1872127Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1872502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5
2023-01-11T20:47:14.1872863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4
2023-01-11T20:47:14.1873347Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1873854Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1874348Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1874734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2
2023-01-11T20:47:14.1875242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1875757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1876142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3
2023-01-11T20:47:14.1876626Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.1877014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0
2023-01-11T20:47:14.1877363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1
2023-01-11T20:47:14.1877726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6
2023-01-11T20:47:14.1878091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7
2023-01-11T20:47:14.1878577Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1879072Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1879573Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1880079Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1880588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1881081Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1881588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1882091Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.1882375Z ok (3.138s)
2023-01-11T20:47:14.1882687Z   test_all_to_all_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1472
2023-01-11T20:47:14.1883076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1473
2023-01-11T20:47:14.1883399Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1474
2023-01-11T20:47:14.1883708Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1475
2023-01-11T20:47:14.1884100Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1476
2023-01-11T20:47:14.1884419Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1477
2023-01-11T20:47:14.1884737Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1478
2023-01-11T20:47:14.1885042Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1479
2023-01-11T20:47:14.1885495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1885833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1886249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1886600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1886920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.1887379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1887735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1888190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1888533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1888839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.1889301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1889626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1890054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1890381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1890704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.1891167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1891492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1891904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1892246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1892565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.1893011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1893347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1893776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1894125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1894439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.1894897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1895228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1895642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1895987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1896309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.1896772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1897091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1897524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1897870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1898195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.1898643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1898970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1899400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1899731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1900051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.1900448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.1900845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.1901194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.1901555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.1902111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.1902473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.1902819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.1903231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.1903730Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1904244Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1904764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1905276Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1905787Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1906285Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1906790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1907305Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1907589Z ok (2.842s)
2023-01-11T20:47:14.1907897Z   test_all_to_all_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1584
2023-01-11T20:47:14.1908287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1585
2023-01-11T20:47:14.1908608Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1586
2023-01-11T20:47:14.1908929Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1587
2023-01-11T20:47:14.1909233Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1588
2023-01-11T20:47:14.1909543Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1589
2023-01-11T20:47:14.1909862Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1590
2023-01-11T20:47:14.1910162Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1591
2023-01-11T20:47:14.1910608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1910935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1911362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1911693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1912015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.1912472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1912790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1913286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1913668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1913990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.1914436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1914763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1915195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1915536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1915840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.1916299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1916624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1917036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1917375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1917695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.1918154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1918462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1918886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1919224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1919536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.1919998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1920322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1920745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1921073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1921393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.1921843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1922167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1922581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1922922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1923241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.1923682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.1924088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.1924514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.1924855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.1925161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.1925525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.1925929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.1926303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.1926660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.1927015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.1927369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.1927711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.1928069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.1928559Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1929076Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1929578Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1955252Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1955841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T20:47:14.1956304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T20:47:14.1956668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T20:47:14.1957181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1957714Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1958242Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1958744Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.1959140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:47:14.1959508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:47:14.1959865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:47:14.1960230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:47:14.1960598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T20:47:14.1961089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1961592Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1962112Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1962501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:47:14.1962867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T20:47:14.1963221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:47:14.1963708Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1964524Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1964920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T20:47:14.1965394Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1965782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T20:47:14.1966266Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1966767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.1967163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:47:14.1967534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:47:14.1967901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T20:47:14.1968374Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1968885Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1969279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T20:47:14.1969644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:47:14.1970116Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1970631Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1971027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T20:47:14.1971512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1972013Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1972517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1973028Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.1973420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:47:14.1973775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T20:47:14.1974132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:47:14.1974495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:47:14.1974844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T20:47:14.1975334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1975848Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1976360Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1976768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:47:14.1977161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T20:47:14.1977653Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1978166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1978669Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1979181Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1979568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:47:14.1980051Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.1980433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T20:47:14.1980795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T20:47:14.1981162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:47:14.1981529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:47:14.1982041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T20:47:14.1982532Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1982924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2023-01-11T20:47:14.1983398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1983917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1984427Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1984942Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1985436Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1985940Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1986325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T20:47:14.1986691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2023-01-11T20:47:14.1987164Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.1987550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2023-01-11T20:47:14.1987903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2023-01-11T20:47:14.1988262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T20:47:14.1988608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T20:47:14.1988955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T20:47:14.1989434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1990038Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1990419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2023-01-11T20:47:14.1990774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T20:47:14.1991255Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1991630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2023-01-11T20:47:14.1992116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1992628Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1993142Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1993516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2023-01-11T20:47:14.1993996Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1994381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2023-01-11T20:47:14.1994863Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.1995244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2023-01-11T20:47:14.1995602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T20:47:14.1995968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2023-01-11T20:47:14.1996454Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1996953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1997340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3
2023-01-11T20:47:14.1997700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2023-01-11T20:47:14.1998184Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1998554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7
2023-01-11T20:47:14.1999034Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1999550Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.1999940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5
2023-01-11T20:47:14.2000410Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2000912Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2001414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2001784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2
2023-01-11T20:47:14.2002149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2023-01-11T20:47:14.2002549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4
2023-01-11T20:47:14.2003064Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2003440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6
2023-01-11T20:47:14.2004000Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2004387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4
2023-01-11T20:47:14.2004872Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2005250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6
2023-01-11T20:47:14.2005737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2006128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3
2023-01-11T20:47:14.2006613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2006993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2023-01-11T20:47:14.2007355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2023-01-11T20:47:14.2007838Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2008352Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2008851Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2009240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2
2023-01-11T20:47:14.2009605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5
2023-01-11T20:47:14.2009957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7
2023-01-11T20:47:14.2010444Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2010951Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2011465Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2011846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5
2023-01-11T20:47:14.2012338Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2012733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7
2023-01-11T20:47:14.2013218Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2013713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2014218Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2014609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6
2023-01-11T20:47:14.2015012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2
2023-01-11T20:47:14.2015518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2015916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4
2023-01-11T20:47:14.2016282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3
2023-01-11T20:47:14.2016649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0
2023-01-11T20:47:14.2017002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1
2023-01-11T20:47:14.2017489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2018006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2018394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1
2023-01-11T20:47:14.2018760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0
2023-01-11T20:47:14.2019248Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2019644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5
2023-01-11T20:47:14.2020117Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2020508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7
2023-01-11T20:47:14.2020999Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2021514Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2022168Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2022560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4
2023-01-11T20:47:14.2023051Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2023447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6
2023-01-11T20:47:14.2023797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2
2023-01-11T20:47:14.2024162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3
2023-01-11T20:47:14.2024653Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2025171Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2025675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2026063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2
2023-01-11T20:47:14.2026553Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2026933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3
2023-01-11T20:47:14.2027304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0
2023-01-11T20:47:14.2027664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6
2023-01-11T20:47:14.2028250Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2028753Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2029148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7
2023-01-11T20:47:14.2029639Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2030149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2030528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1
2023-01-11T20:47:14.2030892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4
2023-01-11T20:47:14.2031253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5
2023-01-11T20:47:14.2031742Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2032243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2032751Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2033261Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2033771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2034155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5
2023-01-11T20:47:14.2034525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1
2023-01-11T20:47:14.2035019Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2035411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3
2023-01-11T20:47:14.2035760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4
2023-01-11T20:47:14.2036119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0
2023-01-11T20:47:14.2036606Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2037104Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2037498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2
2023-01-11T20:47:14.2037673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6
2023-01-11T20:47:14.2037845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7
2023-01-11T20:47:14.2038140Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2038435Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2038726Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2039018Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2039327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2039686Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2039987Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2040278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2040350Z ok (3.138s)
2023-01-11T20:47:14.2040583Z   test_broadcast_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1768
2023-01-11T20:47:14.2040740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1769
2023-01-11T20:47:14.2040896Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1770
2023-01-11T20:47:14.2041053Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1771
2023-01-11T20:47:14.2041192Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1772
2023-01-11T20:47:14.2041343Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1773
2023-01-11T20:47:14.2041498Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1774
2023-01-11T20:47:14.2041650Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1775
2023-01-11T20:47:14.2041927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2042054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2042340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2042476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2042632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.2042908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2043033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2043313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2043449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2043614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.2043963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2044089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2044372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2044496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2044662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.2044931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2045054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2045331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2045466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2045628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.2045897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2046044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2046369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2046506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2046670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.2046936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2047057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2047333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2047464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2047628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.2047884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2048008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2048285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2048417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2048578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.2048845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2048967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2049239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2049375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2049529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.2049712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.2049891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.2050068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.2050241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.2050415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.2050587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.2050758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.2050916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.2051219Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2051513Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2051808Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2052105Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2052396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2052724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2053046Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2053340Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2053408Z ok (2.836s)
2023-01-11T20:47:14.2053623Z   test_broadcast_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1880
2023-01-11T20:47:14.2053780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1881
2023-01-11T20:47:14.2053937Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1882
2023-01-11T20:47:14.2054087Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1883
2023-01-11T20:47:14.2054241Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1884
2023-01-11T20:47:14.2054394Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1885
2023-01-11T20:47:14.2054542Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1886
2023-01-11T20:47:14.2054690Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1887
2023-01-11T20:47:14.2054957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2055082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2055365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2055503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2055667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.2055944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2056069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2056350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2056487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2056638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.2056909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2057032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2057310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2057446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2057611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.2057879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2058001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2058263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2058399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2058559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.2058827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2058981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2059255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2059413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2059577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.2059848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2059956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2060228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2060359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2060521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.2060789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2060912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2061184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2061316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2061464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.2061886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2062011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2062290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2062427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2062587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.2062772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.2062947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.2063122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.2063283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.2063454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.2063622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.2063793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.2064098Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2064273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.2064573Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2064873Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2065167Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2065444Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2065617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T20:47:14.2065841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T20:47:14.2066046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T20:47:14.2066220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:47:14.2066515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2066809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2067102Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2067274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:47:14.2067435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:47:14.2067610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:47:14.2067902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2068071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T20:47:14.2068360Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2068658Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2068830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:47:14.2069007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T20:47:14.2069179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:47:14.2069468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2069747Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2069920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:47:14.2070209Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2070497Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2070784Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2070957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T20:47:14.2071128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T20:47:14.2071297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:47:14.2071466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T20:47:14.2071751Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2072043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2072334Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2072651Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2072851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:47:14.2073023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T20:47:14.2073316Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2073486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:47:14.2073774Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2074070Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2074233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T20:47:14.2074405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T20:47:14.2074576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:47:14.2074860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2075029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:47:14.2075202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T20:47:14.2075501Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2075796Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2076094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2076368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2076659Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2076830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T20:47:14.2077000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T20:47:14.2077296Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2077467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T20:47:14.2077766Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2077935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:47:14.2078106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:47:14.2078385Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2078556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:47:14.2078724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:47:14.2078893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T20:47:14.2079215Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2079539Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2079711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2023-01-11T20:47:14.2079884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T20:47:14.2080176Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2080450Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2080737Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2081032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2081327Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2081617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2081788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2023-01-11T20:47:14.2081964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2023-01-11T20:47:14.2082136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T20:47:14.2082305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2023-01-11T20:47:14.2082477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T20:47:14.2082635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T20:47:14.2082929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2083219Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2083389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T20:47:14.2083558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2023-01-11T20:47:14.2083936Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2084237Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2084527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2084815Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2085094Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2085266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2023-01-11T20:47:14.2085554Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2085725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2023-01-11T20:47:14.2085897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2023-01-11T20:47:14.2086106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2023-01-11T20:47:14.2086302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T20:47:14.2086600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2086770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2023-01-11T20:47:14.2087057Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2087212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2023-01-11T20:47:14.2087381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3
2023-01-11T20:47:14.2087666Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2087961Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2088250Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2088422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5
2023-01-11T20:47:14.2088711Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2088997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2089167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7
2023-01-11T20:47:14.2089444Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2089623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2023-01-11T20:47:14.2089795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2
2023-01-11T20:47:14.2089964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4
2023-01-11T20:47:14.2090135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6
2023-01-11T20:47:14.2090427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2090713Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2091004Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2091298Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2091458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4
2023-01-11T20:47:14.2091629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6
2023-01-11T20:47:14.2091801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2023-01-11T20:47:14.2091971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2
2023-01-11T20:47:14.2092262Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2092554Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2092878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2093208Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2093383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3
2023-01-11T20:47:14.2093556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2023-01-11T20:47:14.2093712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5
2023-01-11T20:47:14.2094003Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2094174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7
2023-01-11T20:47:14.2094467Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2094760Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2095049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2095225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5
2023-01-11T20:47:14.2095515Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2095690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6
2023-01-11T20:47:14.2095848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7
2023-01-11T20:47:14.2096017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2
2023-01-11T20:47:14.2096315Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2096605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2096899Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2097076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3
2023-01-11T20:47:14.2097248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4
2023-01-11T20:47:14.2097417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0
2023-01-11T20:47:14.2097586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1
2023-01-11T20:47:14.2097872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2098173Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2098471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2098642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1
2023-01-11T20:47:14.2098814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0
2023-01-11T20:47:14.2099105Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2099399Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2099725Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2100047Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2100340Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2100502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5
2023-01-11T20:47:14.2100674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4
2023-01-11T20:47:14.2100845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7
2023-01-11T20:47:14.2101015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6
2023-01-11T20:47:14.2101185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2
2023-01-11T20:47:14.2101356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3
2023-01-11T20:47:14.2101806Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2102118Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2102412Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2102569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3
2023-01-11T20:47:14.2102865Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2103044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2
2023-01-11T20:47:14.2103338Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2103632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2103926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2104217Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2104391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7
2023-01-11T20:47:14.2104563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1
2023-01-11T20:47:14.2104729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6
2023-01-11T20:47:14.2104885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0
2023-01-11T20:47:14.2105057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4
2023-01-11T20:47:14.2105227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5
2023-01-11T20:47:14.2105520Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2105814Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2106105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2106400Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2106635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5
2023-01-11T20:47:14.2106964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2107243Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2107414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1
2023-01-11T20:47:14.2107585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0
2023-01-11T20:47:14.2107752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4
2023-01-11T20:47:14.2108045Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2108342Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2108518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3
2023-01-11T20:47:14.2108689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2
2023-01-11T20:47:14.2108855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6
2023-01-11T20:47:14.2109009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7
2023-01-11T20:47:14.2109302Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2109594Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2109889Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2110185Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2110476Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2110764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2111056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2111345Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2111412Z ok (3.139s)
2023-01-11T20:47:14.2111635Z   test_reduce_scatter_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2064
2023-01-11T20:47:14.2111797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2065
2023-01-11T20:47:14.2111955Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2066
2023-01-11T20:47:14.2112107Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2067
2023-01-11T20:47:14.2112257Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 2068
2023-01-11T20:47:14.2112404Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 2069
2023-01-11T20:47:14.2112550Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 2070
2023-01-11T20:47:14.2112696Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 2071
2023-01-11T20:47:14.2112961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2113119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2113433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2113568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2113730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.2114002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2114123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2114407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2114541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2114691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.2114959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2115079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2115353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2115484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2115643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.2115908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2116029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2116290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2116426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2116588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.2116853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2116974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2117248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2117379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2117543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.2117806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2117915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2118190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2118326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2118487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.2118754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2118877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2119154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2119286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2119434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.2119734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2119881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2120157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2120291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2120450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.2120629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.2120806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.2120977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.2121134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.2121305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.2121478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.2121653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.2121821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.2122120Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2122417Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2122707Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2123004Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2123289Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2123582Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2123955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2124247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2124626Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2124790Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2125168Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2125325Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2125703Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2125860Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2126216Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2126372Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2126817Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2126974Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2127337Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2127490Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2127856Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2128009Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2128382Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2128539Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2128595Z ok (2.738s)
2023-01-11T20:47:14.2128832Z   test_reduce_scatter_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2176
2023-01-11T20:47:14.2128987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2177
2023-01-11T20:47:14.2129138Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2178
2023-01-11T20:47:14.2129289Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2179
2023-01-11T20:47:14.2129437Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 2180
2023-01-11T20:47:14.2129587Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 2181
2023-01-11T20:47:14.2129738Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 2182
2023-01-11T20:47:14.2129881Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 2183
2023-01-11T20:47:14.2130155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2130281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2130561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2130697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2130860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.2131129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2131253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2131532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2131653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2131814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.2132084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2132205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2132484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2132618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2132782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.2133095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2133234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2133519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2133654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2133815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.2134086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2134207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2134482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2134620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2134780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.2135039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2135162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2135438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2135573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2135732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.2136000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2136120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2136396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2136528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2136677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.2136943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2137063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2137334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2137464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2137631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.2137813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.2137992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.2138156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.2138328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.2138500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.2138669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.2138836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.2139003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.2139340Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2139684Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2139981Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2140265Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2140555Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2140731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T20:47:14.2140905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:47:14.2141080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T20:47:14.2141252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T20:47:14.2141546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2141966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2142267Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2142439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:47:14.2142600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:47:14.2142770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:47:14.2142943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T20:47:14.2143243Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2143537Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2143829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2144117Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2144409Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2144698Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2144858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T20:47:14.2145038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T20:47:14.2145210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:47:14.2145380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:47:14.2145551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T20:47:14.2145840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2146135Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2146395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:47:14.2146599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:47:14.2146757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T20:47:14.2147049Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2147340Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2147509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T20:47:14.2147677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:47:14.2147966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2148263Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2148549Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2148837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2149008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T20:47:14.2149284Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2149572Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2149742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:47:14.2149915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T20:47:14.2150087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:47:14.2150256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:47:14.2150425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T20:47:14.2150713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2151004Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2151162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:47:14.2151335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T20:47:14.2151630Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2151922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2152215Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2152505Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2152676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:47:14.2152967Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2153289Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2153488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T20:47:14.2153647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T20:47:14.2153816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:47:14.2153988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:47:14.2154156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T20:47:14.2154444Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2154734Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2155028Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2155321Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2155494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2023-01-11T20:47:14.2155770Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2155938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T20:47:14.2156106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2023-01-11T20:47:14.2156275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2023-01-11T20:47:14.2156566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2156860Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2157032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T20:47:14.2157200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2023-01-11T20:47:14.2157492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2157650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T20:47:14.2157940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2158111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T20:47:14.2158401Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2158690Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2158860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T20:47:14.2159029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2023-01-11T20:47:14.2159199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2023-01-11T20:47:14.2159487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2159805Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2160022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2023-01-11T20:47:14.2160311Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2160595Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2160769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2023-01-11T20:47:14.2161053Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2161226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2023-01-11T20:47:14.2161396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T20:47:14.2161569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2023-01-11T20:47:14.2161867Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2162140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2162308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3
2023-01-11T20:47:14.2162478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2023-01-11T20:47:14.2162765Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2162934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5
2023-01-11T20:47:14.2163224Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2163518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2163867Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2164163Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2164442Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2164614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7
2023-01-11T20:47:14.2164783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2023-01-11T20:47:14.2164956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2
2023-01-11T20:47:14.2165125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4
2023-01-11T20:47:14.2165293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6
2023-01-11T20:47:14.2165588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2165882Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2166172Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2166343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6
2023-01-11T20:47:14.2166536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2023-01-11T20:47:14.2166737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4
2023-01-11T20:47:14.2167032Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2167322Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2167614Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2167905Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2168074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2023-01-11T20:47:14.2168245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2
2023-01-11T20:47:14.2168539Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2168696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3
2023-01-11T20:47:14.2168863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5
2023-01-11T20:47:14.2169033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7
2023-01-11T20:47:14.2169325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2169615Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2169904Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2170192Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2170484Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2170659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7
2023-01-11T20:47:14.2170818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5
2023-01-11T20:47:14.2170986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3
2023-01-11T20:47:14.2171153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4
2023-01-11T20:47:14.2171443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2171736Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2172026Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2172195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2
2023-01-11T20:47:14.2172372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6
2023-01-11T20:47:14.2172542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0
2023-01-11T20:47:14.2172711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1
2023-01-11T20:47:14.2172996Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2173295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2173650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2173824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4
2023-01-11T20:47:14.2173992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1
2023-01-11T20:47:14.2174159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0
2023-01-11T20:47:14.2174452Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2174746Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2175037Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2175320Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2175613Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2175786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6
2023-01-11T20:47:14.2175954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7
2023-01-11T20:47:14.2176122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5
2023-01-11T20:47:14.2176288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2
2023-01-11T20:47:14.2176452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3
2023-01-11T20:47:14.2176748Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2177039Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2177335Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2177615Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2177904Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2178194Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2178366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3
2023-01-11T20:47:14.2178663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2178951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2179125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2
2023-01-11T20:47:14.2179294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7
2023-01-11T20:47:14.2179460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6
2023-01-11T20:47:14.2179613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1
2023-01-11T20:47:14.2179781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0
2023-01-11T20:47:14.2179980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4
2023-01-11T20:47:14.2180173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5
2023-01-11T20:47:14.2180469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2180762Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2181054Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2181223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0
2023-01-11T20:47:14.2181388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4
2023-01-11T20:47:14.2181542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5
2023-01-11T20:47:14.2182020Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2182321Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2182492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3
2023-01-11T20:47:14.2182786Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2182955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2
2023-01-11T20:47:14.2183246Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2183539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2183710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1
2023-01-11T20:47:14.2183879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6
2023-01-11T20:47:14.2184041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7
2023-01-11T20:47:14.2184337Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2184629Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2184921Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2185212Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2185505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2185795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2186086Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2186375Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2186743Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2186905Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2187348Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2187557Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2187933Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2188091Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2188458Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2188617Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2188993Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2189153Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2189522Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2189664Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2190031Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2190186Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2190558Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2190715Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2190786Z ok (3.138s)
2023-01-11T20:47:14.2191027Z   test_reduce_scatter_uneven (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2360
2023-01-11T20:47:14.2191182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2361
2023-01-11T20:47:14.2191336Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2362
2023-01-11T20:47:14.2191476Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2363
2023-01-11T20:47:14.2191624Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 2364
2023-01-11T20:47:14.2191772Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 2365
2023-01-11T20:47:14.2191922Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 2366
2023-01-11T20:47:14.2192077Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 2367
2023-01-11T20:47:14.2192357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2192483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2192766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2192890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2193055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.2193328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2193451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2193762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2193897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2194089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.2194362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2194487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2194749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2194881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2195044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.2195314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2195439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2195714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2195850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2196013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.2196266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2196390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2196663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2196795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2196957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.2197227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2197348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2197621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2197755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2197901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.2198167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2198288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2198559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2198693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2198853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.2199118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2199239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2199498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2199632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2199790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.2199968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.2200176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.2200377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.2200550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.2200724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.2200893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.2201049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.2201219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.2201522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2201826Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2202126Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2202418Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2202706Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2202997Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2203290Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2203582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2204021Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2204181Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2204563Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2204722Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2205095Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2205251Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2205629Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2205785Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2206153Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2206307Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2206665Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2206823Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2207187Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2207403Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2207774Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:47:14.2207930Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:47:14.2207999Z ok (2.742s)
2023-01-11T20:47:14.2208225Z   test_scatter_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2472
2023-01-11T20:47:14.2208384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2473
2023-01-11T20:47:14.2208523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2474
2023-01-11T20:47:14.2208679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2475
2023-01-11T20:47:14.2208827Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 2476
2023-01-11T20:47:14.2208981Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 2477
2023-01-11T20:47:14.2209130Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 2478
2023-01-11T20:47:14.2209278Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 2479
2023-01-11T20:47:14.2209559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2209685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2209956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2210095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2210265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.2210541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2210666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2210953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2211089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2211253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.2211528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2211637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2211918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2212057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2212223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.2212496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2212618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2212895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2213029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2213179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.2213448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2213603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2213906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2214042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2214205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.2214474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2214596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2214870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2214991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2215152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.2215425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2215547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2215818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2215951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2216112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.2216382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2216505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2216771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2216909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2217079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.2217256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.2217435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.2217611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.2217786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.2217956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.2218114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.2218286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.2218589Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2218759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.2219057Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2219356Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2219648Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2219940Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2220262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2220591Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2220875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2220948Z ok (2.838s)
2023-01-11T20:47:14.2221172Z   test_scatter_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2584
2023-01-11T20:47:14.2221328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2585
2023-01-11T20:47:14.2221481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2586
2023-01-11T20:47:14.2221634Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2587
2023-01-11T20:47:14.2222002Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 2588
2023-01-11T20:47:14.2222158Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 2589
2023-01-11T20:47:14.2222297Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 2590
2023-01-11T20:47:14.2222449Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 2591
2023-01-11T20:47:14.2222730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2222855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2223139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2223277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2223442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.2223716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2223845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2224113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2224247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2224410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.2224680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2224803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2225083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2225220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2225381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.2225639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2225762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2226038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2226169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2226328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.2226599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2226723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2227064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2227231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2227379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.2227648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2227769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2228050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2228186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2228353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.2228629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2228753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2229022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2229159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2229322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.2229590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2229712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2229987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2230122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2230285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.2230464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.2230628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.2230801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.2230976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.2231148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.2231320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.2231490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.2231664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.2231962Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2232261Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2232546Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2232847Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2233137Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2233429Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2233636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T20:47:14.2233852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T20:47:14.2234025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:47:14.2234195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:47:14.2234366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T20:47:14.2234649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2234943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2235116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:47:14.2235292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:47:14.2235462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T20:47:14.2235754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2236044Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2236215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:47:14.2236384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T20:47:14.2236665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2236839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:47:14.2237129Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2237423Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2237713Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2238004Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2238294Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2238465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:47:14.2238641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T20:47:14.2238815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T20:47:14.2238975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:47:14.2239144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T20:47:14.2239436Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2239728Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2240017Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2240338Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2240535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T20:47:14.2240706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:47:14.2240875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T20:47:14.2241152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2241442Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2241613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:47:14.2241782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T20:47:14.2242080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2242372Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2242543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:47:14.2242716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:47:14.2242884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T20:47:14.2243175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2243454Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2243628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:47:14.2243872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T20:47:14.2244169Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2244461Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2244753Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2245039Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2245209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:47:14.2245382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T20:47:14.2245659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2245954Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2246127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:47:14.2246298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T20:47:14.2246467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:47:14.2246637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T20:47:14.2246931Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2247319Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2247490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2023-01-11T20:47:14.2247675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T20:47:14.2247980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2248271Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2248442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2023-01-11T20:47:14.2248732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2249028Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2249199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T20:47:14.2249373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2023-01-11T20:47:14.2249663Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2249953Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2250111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2023-01-11T20:47:14.2250282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T20:47:14.2250454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T20:47:14.2250745Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2251037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2251326Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2251495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T20:47:14.2251664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2023-01-11T20:47:14.2251955Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2252114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2023-01-11T20:47:14.2252402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2252693Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2252864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2023-01-11T20:47:14.2253150Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2253321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2023-01-11T20:47:14.2253612Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2253816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2023-01-11T20:47:14.2254017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T20:47:14.2254172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2023-01-11T20:47:14.2254463Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2254752Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2254925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3
2023-01-11T20:47:14.2255214Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2255382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5
2023-01-11T20:47:14.2255674Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2255967Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2256138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7
2023-01-11T20:47:14.2256430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2256705Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2256994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2257163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2023-01-11T20:47:14.2257336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2
2023-01-11T20:47:14.2257508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2023-01-11T20:47:14.2257681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4
2023-01-11T20:47:14.2257851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6
2023-01-11T20:47:14.2258140Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2258431Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2258587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6
2023-01-11T20:47:14.2258761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4
2023-01-11T20:47:14.2259057Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2259347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2259633Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2259924Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2260096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2023-01-11T20:47:14.2260384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2260588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3
2023-01-11T20:47:14.2260783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2
2023-01-11T20:47:14.2260942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2023-01-11T20:47:14.2261229Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2261398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5
2023-01-11T20:47:14.2261569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7
2023-01-11T20:47:14.2261985Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2262278Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2262463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5
2023-01-11T20:47:14.2262636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7
2023-01-11T20:47:14.2262926Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2263200Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2263375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6
2023-01-11T20:47:14.2263546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4
2023-01-11T20:47:14.2263837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2264132Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2264424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2264712Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2264886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3
2023-01-11T20:47:14.2265057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2
2023-01-11T20:47:14.2265213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0
2023-01-11T20:47:14.2265383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1
2023-01-11T20:47:14.2265685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2265982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2266278Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2266446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1
2023-01-11T20:47:14.2266613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0
2023-01-11T20:47:14.2266911Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2267206Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2267436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5
2023-01-11T20:47:14.2267625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7
2023-01-11T20:47:14.2267924Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2268220Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2268395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6
2023-01-11T20:47:14.2268566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4
2023-01-11T20:47:14.2268859Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2269033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2
2023-01-11T20:47:14.2269203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3
2023-01-11T20:47:14.2269499Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2269780Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2269955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3
2023-01-11T20:47:14.2270125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2
2023-01-11T20:47:14.2270421Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2270716Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2270892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7
2023-01-11T20:47:14.2271184Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2271477Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2271650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6
2023-01-11T20:47:14.2271929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2272219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2272391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1
2023-01-11T20:47:14.2272568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0
2023-01-11T20:47:14.2272741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4
2023-01-11T20:47:14.2272912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5
2023-01-11T20:47:14.2273209Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2273503Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2273673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5
2023-01-11T20:47:14.2273843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4
2023-01-11T20:47:14.2274153Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2274475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2274772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2274944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1
2023-01-11T20:47:14.2275238Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2275411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0
2023-01-11T20:47:14.2275701Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2275879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3
2023-01-11T20:47:14.2276051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2
2023-01-11T20:47:14.2276333Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2276510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6
2023-01-11T20:47:14.2276680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7
2023-01-11T20:47:14.2276969Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2277261Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2277554Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2277848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2278136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2278428Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2278718Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2278996Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2279065Z ok (3.141s)
2023-01-11T20:47:14.2279300Z   test_scatter_uneven (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2768
2023-01-11T20:47:14.2279460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2769
2023-01-11T20:47:14.2279641Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2770
2023-01-11T20:47:14.2279795Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2771
2023-01-11T20:47:14.2279945Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 2772
2023-01-11T20:47:14.2280096Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 2773
2023-01-11T20:47:14.2280235Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 2774
2023-01-11T20:47:14.2280387Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 2775
2023-01-11T20:47:14.2280669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2280795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2281113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2281289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2281456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.2281730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2281854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2282125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2282261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2282424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.2282699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2282826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2283105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2283241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2283405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.2283664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2283848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2284134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2284267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2284433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.2284705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2284830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2285107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2285240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2285389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.2285656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2285778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2286057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2286193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2286354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.2286620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2286741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2287002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2287136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2287295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.2287559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2287718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2288020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2288155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2288314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.2288492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.2288655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.2288832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.2289005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.2289177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.2289350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.2289522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.2289691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.2289991Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2290291Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2290573Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2290874Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2291167Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2291459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2291751Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2292047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2292118Z ok (2.738s)
2023-01-11T20:47:14.2292328Z   test_device_mesh_2d (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2880
2023-01-11T20:47:14.2292484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2881
2023-01-11T20:47:14.2292626Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2882
2023-01-11T20:47:14.2292782Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2883
2023-01-11T20:47:14.2292932Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 2884
2023-01-11T20:47:14.2293083Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 2885
2023-01-11T20:47:14.2293279Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 2886
2023-01-11T20:47:14.2293429Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 2887
2023-01-11T20:47:14.2293706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2293830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2294101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2294268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2294462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.2294738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2294860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2295140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2295274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2295437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.2295706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2295818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2296096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2296233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2296394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.2296660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2296784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2297057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2297189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2297336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.2297606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2297729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2298005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2298136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2298295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.2298561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2298683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2298964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2299089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2299254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.2299526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2299647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2299926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2300061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2300225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.2300496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2300638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2300915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2301078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2301240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.2301418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.2301597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.2302022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.2302197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.2302369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.2302528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.2302701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.2303005Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2303177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.2303477Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2303646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T20:47:14.2303815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T20:47:14.2304111Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2304408Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2304690Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2304985Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2305275Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2305564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2305735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:47:14.2305909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T20:47:14.2306085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T20:47:14.2306257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:47:14.2306426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:47:14.2306593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:47:14.2306875Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2307165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2307336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:47:14.2307563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:47:14.2307895Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2308194Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2308486Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2308778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2309071Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2309347Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2309522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T20:47:14.2309696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T20:47:14.2309870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T20:47:14.2310037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T20:47:14.2310207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:47:14.2310378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:47:14.2310669Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2310964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2311124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:47:14.2311420Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2311711Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2312001Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2312289Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2312576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2312749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T20:47:14.2312922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T20:47:14.2313213Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2313383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T20:47:14.2313542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:47:14.2313714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T20:47:14.2313880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:47:14.2314049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:47:14.2314371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2314695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2314867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:47:14.2315034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:47:14.2315327Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2315602Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2315896Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2316187Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2316479Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2316650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T20:47:14.2316821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T20:47:14.2317112Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2317283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T20:47:14.2317455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T20:47:14.2317625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:47:14.2317784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:47:14.2318076Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2318369Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2318655Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2318946Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2319235Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2319530Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2319820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2320110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2320165Z ok (2.944s)
2023-01-11T20:47:14.2320391Z   test_device_mesh_2d_from_dim_groups (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3016
2023-01-11T20:47:14.2320548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3017
2023-01-11T20:47:14.2320701Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3018
2023-01-11T20:47:14.2320853Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3019
2023-01-11T20:47:14.2321033Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 3020
2023-01-11T20:47:14.2321181Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 3021
2023-01-11T20:47:14.2321358Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 3022
2023-01-11T20:47:14.2321496Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 3023
2023-01-11T20:47:14.2321775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2321902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2322187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2322326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2322489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.2322762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2322885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2323165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2323309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2323473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.2323741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2323942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2324223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2324356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2324519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.2324788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2324897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2325175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2325308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2325466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.2325735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2325858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2326136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2326273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2326433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.2326695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2326819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2327100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2327235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2327398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.2327670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2327828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2328151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2328285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2328434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.2328711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2328834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2329110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2329236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2329399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.2329589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.2329765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.2329940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.2330110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.2330282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.2330467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.2330622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.2330796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.2331097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2331397Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2331691Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2331984Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2332275Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2332565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2332859Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2333155Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2333314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T20:47:14.2333492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T20:47:14.2333663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T20:47:14.2333834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T20:47:14.2334001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:47:14.2334174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:47:14.2334376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:47:14.2334571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:47:14.2334871Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2335029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:47:14.2335322Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2335491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:47:14.2335779Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2336076Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2336363Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2336532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T20:47:14.2336817Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2337106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2337276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T20:47:14.2337550Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2337724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T20:47:14.2337896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T20:47:14.2338067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:47:14.2338260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:47:14.2338548Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2338835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2339005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:47:14.2339292Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2339570Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2339855Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2340024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T20:47:14.2340306Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2340591Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2340760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T20:47:14.2340932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T20:47:14.2341135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:47:14.2341449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2341607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T20:47:14.2341906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:47:14.2342075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:47:14.2342363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2342650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2342822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:47:14.2342993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:47:14.2343287Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2343576Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2343744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T20:47:14.2344019Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2344315Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2344612Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2344904Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2345077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T20:47:14.2345248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T20:47:14.2345418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T20:47:14.2345586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:47:14.2345755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:47:14.2346033Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2346328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2346615Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2346899Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2347185Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2347479Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2347766Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2348106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2348171Z ok (2.939s)
2023-01-11T20:47:14.2348418Z   test_device_mesh_dim_groups_error (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3152
2023-01-11T20:47:14.2348575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3153
2023-01-11T20:47:14.2348729Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3154
2023-01-11T20:47:14.2348882Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3155
2023-01-11T20:47:14.2349034Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 3156
2023-01-11T20:47:14.2349187Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 3157
2023-01-11T20:47:14.2349338Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 3158
2023-01-11T20:47:14.2349488Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 3159
2023-01-11T20:47:14.2349767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2349878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2350165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2350303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2350466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.2350737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2350858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2351134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2351269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2351419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.2351689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2351809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2352084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2352218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2352378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.2352644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2352767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2353043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2353164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2353322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.2353587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2353706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2353979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2354110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2354267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.2354619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2354756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2355039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2355174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2355340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.2355612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2355734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2356017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2356153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2356319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.2356575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2356696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2356971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2357103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2357265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.2357441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.2357615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.2357789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.2357963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.2358119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.2358288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.2358456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.2358626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.2358924Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2359096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T20:47:14.2359393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2359690Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2359984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2360261Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2360431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:47:14.2360598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:47:14.2360883Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2361235Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2361527Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2361698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T20:47:14.2361868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T20:47:14.2362038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T20:47:14.2362191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:47:14.2362482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2362654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:47:14.2362949Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2363117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:47:14.2363285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:47:14.2363572Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2363954Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2364248Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2364408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T20:47:14.2364697Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2364868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T20:47:14.2365039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T20:47:14.2365209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T20:47:14.2365502Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2365786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2365959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:47:14.2366132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:47:14.2366422Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2366697Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2366989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2367162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:47:14.2367332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:47:14.2367619Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2367972Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2368262Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2368431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T20:47:14.2368721Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2368875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T20:47:14.2369045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T20:47:14.2369214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T20:47:14.2369508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2369680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:47:14.2369848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:47:14.2370136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2370423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2370591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:47:14.2370747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:47:14.2371042Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2371336Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2371506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T20:47:14.2371794Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2372085Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2372255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T20:47:14.2372428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T20:47:14.2372595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T20:47:14.2372891Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2373167Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2373336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:47:14.2373517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:47:14.2373809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2374100Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2374388Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2374747Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2375037Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2375321Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2375605Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2375879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2375960Z ok (2.840s)
2023-01-11T20:47:14.2376170Z   test_device_mesh_nd (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3288
2023-01-11T20:47:14.2376327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3289
2023-01-11T20:47:14.2376480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3290
2023-01-11T20:47:14.2376631Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3291
2023-01-11T20:47:14.2376776Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 3292
2023-01-11T20:47:14.2376928Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 3293
2023-01-11T20:47:14.2377063Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 3294
2023-01-11T20:47:14.2377210Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 3295
2023-01-11T20:47:14.2377484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2377610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2377898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2378036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2378199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:14.2378471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2378581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2378857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2378988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2379149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:47:14.2379420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2379541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2379818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2379949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2380108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:14.2380360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2380480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2380752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2380884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2381115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:47:14.2381410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2381531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2381962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2382080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2382240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:14.2382508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2382630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2382909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2383045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2383204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:47:14.2383471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2383591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2383850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2383982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2384140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:47:14.2384407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:14.2384529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:14.2384805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:14.2384970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:14.2385129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:47:14.2385291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:14.2385467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:14.2385639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:47:14.2385810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:14.2385981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:47:14.2386151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:47:14.2386320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:47:14.2386617Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2386783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:47:14.2387067Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2387361Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2387701Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2388027Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2388204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:47:14.2388374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T20:47:14.2388546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T20:47:14.2388711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T20:47:14.2389007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2389299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2389579Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:47:14.2389747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:47:14.2389919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:47:14.2390088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:47:14.2390258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T20:47:14.2390549Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2390848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2391022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T20:47:14.2391191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:47:14.2391466Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2391754Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2392042Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2392329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2392617Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2392909Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:47:14.2393082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:47:14.2393253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:47:14.2393422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T20:47:14.2393592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T20:47:14.2393748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:47:14.2393922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T20:47:14.2394214Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2394572Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2394742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:47:14.2394913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T20:47:14.2395202Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2395491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2395777Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2395935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T20:47:14.2396229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2396521Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2396808Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:47:14.2396977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:47:14.2397149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:47:14.2397319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T20:47:14.2397486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:47:14.2397658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T20:47:14.2397931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2398225Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2398396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:47:14.2398565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T20:47:14.2398855Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2399146Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2399319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:47:14.2399489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T20:47:14.2399778Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2400073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2400231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T20:47:14.2400526Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2400814Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:47:14.2400985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:47:14.2401192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:47:14.2401509Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2401681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T20:47:14.2401971Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2402142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T20:47:14.2402297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2023-01-11T20:47:14.2402589Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2402881Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2403056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2023-01-11T20:47:14.2403347Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2403634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2403878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2023-01-11T20:47:14.2404167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2404453Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:47:14.2404627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T20:47:14.2404788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2023-01-11T20:47:14.2404957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T20:47:14.2405127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T20:47:14.2405415Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2405705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2405874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2023-01-11T20:47:14.2406041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T20:47:14.2406337Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2406505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2023-01-11T20:47:14.2406778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2407071Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2407360Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2407648Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2407857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2023-01-11T20:47:14.2408027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2023-01-11T20:47:14.2408344Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:47:14.2408518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2023-01-11T20:47:14.2408687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T20:47:14.2408842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2023-01-11T20:47:14.2409130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2409418Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2409590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2023-01-11T20:47:14.2409759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3
2023-01-11T20:47:14.2410048Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2410337Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2410625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2410793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2
2023-01-11T20:47:14.2410949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2023-01-11T20:47:14.2411237Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2411526Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2411811Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:47:14.2411977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5
2023-01-11T20:47:14.2412146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7
2023-01-11T20:47:14.2412314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4
2023-01-11T20:47:14.2412483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6
2023-01-11T20:47:14.2412770Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2413062Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2413218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6
2023-01-11T20:47:14.2413383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4
2023-01-11T20:47:14.2413669Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2413957Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2414241Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2414525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2414753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3
2023-01-11T20:47:14.2414920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2023-01-11T20:47:14.2415208Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2415363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2
2023-01-11T20:47:14.2415649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:47:14.2415816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2023-01-11T20:47:14.2415984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5
2023-01-11T20:47:14.2416157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7
2023-01-11T20:47:14.2416448Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2416733Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2416908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7
2023-01-11T20:47:14.2417076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5
2023-01-11T20:47:14.2417367Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2417527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2
2023-01-11T20:47:14.2417818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2418108Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2418396Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2418569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6
2023-01-11T20:47:14.2418737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4
2023-01-11T20:47:14.2419025Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2419313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:47:14.2419488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3
2023-01-11T20:47:14.2419646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0
2023-01-11T20:47:14.2419941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2420115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1
2023-01-11T20:47:14.2420411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2420583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0
2023-01-11T20:47:14.2420752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1
2023-01-11T20:47:14.2421047Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2421374Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2421835Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2421997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7
2023-01-11T20:47:14.2422165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5
2023-01-11T20:47:14.2422463Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2422750Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2423042Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:47:14.2423216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6
2023-01-11T20:47:14.2423387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4
2023-01-11T20:47:14.2423560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2
2023-01-11T20:47:14.2423728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3
2023-01-11T20:47:14.2424022Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2424301Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2424593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2424765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3
2023-01-11T20:47:14.2424936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2
2023-01-11T20:47:14.2425229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2425403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0
2023-01-11T20:47:14.2425639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1
2023-01-11T20:47:14.2425934Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2426223Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2426400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6
2023-01-11T20:47:14.2426694Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2426972Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:47:14.2427144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7
2023-01-11T20:47:14.2427316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4
2023-01-11T20:47:14.2427485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5
2023-01-11T20:47:14.2427781Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2428073Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2428310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4
2023-01-11T20:47:14.2428476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5
2023-01-11T20:47:14.2428769Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2429048Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2429354Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2429644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2429936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2430112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3
2023-01-11T20:47:14.2430285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2
2023-01-11T20:47:14.2430456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0
2023-01-11T20:47:14.2430622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1
2023-01-11T20:47:14.2430914Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:47:14.2431071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6
2023-01-11T20:47:14.2431247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7
2023-01-11T20:47:14.2431542Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2431836Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2432128Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2432418Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2432706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2432998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2433286Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2433584Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:47:14.2433639Z ok (3.141s)
2023-01-11T20:47:14.2433650Z 
2023-01-11T20:47:14.2433849Z ----------------------------------------------------------------------
2023-01-11T20:47:14.2433928Z Ran 19 tests in 56.118s
2023-01-11T20:47:14.2433934Z 
2023-01-11T20:47:14.2433995Z OK
2023-01-11T20:47:14.2434000Z 
2023-01-11T20:47:14.2434084Z Generating XML reports...
2023-01-11T20:47:14.2434441Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshCollectiveTest-20230111204617.xml
2023-01-11T20:47:14.2434763Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshTest-20230111204617.xml
2023-01-11T20:47:14.2434768Z 
2023-01-11T20:47:14.2435136Z ##[endgroup]
2023-01-11T20:47:14.2435534Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_device_mesh (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_device_mesh_ww9x5n7u)
2023-01-11T20:47:14.2435540Z 
2023-01-11T20:47:14.2435802Z Running distributed/algorithms/quantization/test_quantization ... [2023-01-11 20:47:14.162302]
2023-01-11T20:47:14.2435971Z MPI not available -- MPI backend tests will be skipped
2023-01-11T20:47:14.2436263Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2}
2023-01-11T20:47:14.2436418Z Running distributed tests for the test backend with env init_method in shard 1 of 2
2023-01-11T20:47:14.2436839Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:47:14.169227]
2023-01-11T20:47:15.6639321Z 
2023-01-11T20:47:15.6639815Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T20:47:15.6640753Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_fvjowpmg)
2023-01-11T20:47:15.6641198Z <unittest.suite.TestSuite tests=[]>
2023-01-11T20:47:15.6641409Z 
2023-01-11T20:47:15.6641661Z ##[endgroup]
2023-01-11T20:47:15.6642358Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_fvjowpmg)
2023-01-11T20:47:15.6642679Z 
2023-01-11T20:47:15.6650485Z Running distributed tests for the test backend with file init_method in shard 1 of 2
2023-01-11T20:47:15.6654897Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:47:15.665276]
2023-01-11T20:47:17.1420826Z 
2023-01-11T20:47:17.1421499Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T20:47:17.1422841Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_y4_i3hm2)
2023-01-11T20:47:17.1423632Z <unittest.suite.TestSuite tests=[]>
2023-01-11T20:47:17.1423810Z 
2023-01-11T20:47:17.1424047Z ##[endgroup]
2023-01-11T20:47:17.1425032Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_y4_i3hm2)
2023-01-11T20:47:17.1425439Z 
2023-01-11T20:47:17.1431104Z Running distributed tests for the gloo backend with env init_method in shard 1 of 2
2023-01-11T20:47:17.1435781Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:47:17.143339]
2023-01-11T20:47:31.8553662Z 
2023-01-11T20:47:31.8554372Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T20:47:31.8555827Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_8jiywey0)
2023-01-11T20:47:31.8557196Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2023-01-11T20:47:31.8558285Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2023-01-11T20:47:31.8558760Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2023-01-11T20:47:31.8559027Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2023-01-11T20:47:31.8559350Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2023-01-11T20:47:31.8559612Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2023-01-11T20:47:31.8559892Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2023-01-11T20:47:31.8560149Z <unittest.suite.TestSuite tests=[]>
2023-01-11T20:47:31.8560662Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T20:47:31.8561195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:31.8561532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:31.8561968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:31.8562320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:31.8562485Z 
2023-01-11T20:47:31.8562558Z Running tests...
2023-01-11T20:47:31.8562869Z ----------------------------------------------------------------------
2023-01-11T20:47:31.8563249Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3514
2023-01-11T20:47:31.8563644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3515
2023-01-11T20:47:31.8569494Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3516
2023-01-11T20:47:31.8570523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:31.8571182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:31.8571953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:31.8572319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:31.8572662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:31.8573117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:31.8573454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:31.8573881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:31.8574228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:31.8574539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:31.8575002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:31.8575328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:31.8575743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:31.8576089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:31.8576408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:31.8576772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:31.8577125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:31.8577620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T20:47:31.8578016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:31.8578508Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T20:47:31.8579166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T20:47:31.8579459Z ok (1.980s)
2023-01-11T20:47:31.8579566Z 
2023-01-11T20:47:31.8579770Z ----------------------------------------------------------------------
2023-01-11T20:47:31.8580001Z Ran 1 test in 1.981s
2023-01-11T20:47:31.8580112Z 
2023-01-11T20:47:31.8580176Z OK
2023-01-11T20:47:31.8580264Z 
2023-01-11T20:47:31.8580349Z Generating XML reports...
2023-01-11T20:47:31.8580860Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204719.xml
2023-01-11T20:47:31.8581439Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T20:47:31.8582119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:31.8582460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:31.8582886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:31.8583236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:31.8583403Z 
2023-01-11T20:47:31.8583480Z Running tests...
2023-01-11T20:47:31.8583789Z ----------------------------------------------------------------------
2023-01-11T20:47:31.8584169Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3567
2023-01-11T20:47:31.8584558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3568
2023-01-11T20:47:31.8584889Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3569
2023-01-11T20:47:31.8585334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:31.8585693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:31.8586125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:31.8586471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:31.8586782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:31.8623577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:31.8625369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:31.8625839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:31.8626192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:31.8626520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:31.8626987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:31.8627314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:31.8627743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:31.8628076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:31.8628400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:31.8628975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:31.8629513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:31.8629866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:31.8630594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T20:47:31.8631126Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T20:47:31.8631627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T20:47:31.8631916Z ok (1.973s)
2023-01-11T20:47:31.8632024Z 
2023-01-11T20:47:31.8632227Z ----------------------------------------------------------------------
2023-01-11T20:47:31.8632471Z Ran 1 test in 1.973s
2023-01-11T20:47:31.8632570Z 
2023-01-11T20:47:31.8632633Z OK
2023-01-11T20:47:31.8632725Z 
2023-01-11T20:47:31.8632811Z Generating XML reports...
2023-01-11T20:47:31.8636192Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204723.xml
2023-01-11T20:47:31.8636840Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T20:47:31.8637511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:31.8638003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:31.8638713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:31.8639343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:31.8639634Z 
2023-01-11T20:47:31.8639768Z Running tests...
2023-01-11T20:47:31.8640268Z ----------------------------------------------------------------------
2023-01-11T20:47:31.8640610Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:47:31.8640823Z 
2023-01-11T20:47:31.8641022Z ----------------------------------------------------------------------
2023-01-11T20:47:31.8641260Z Ran 1 test in 0.001s
2023-01-11T20:47:31.8641375Z 
2023-01-11T20:47:31.8641450Z OK (skipped=1)
2023-01-11T20:47:31.8641546Z 
2023-01-11T20:47:31.8641630Z Generating XML reports...
2023-01-11T20:47:31.8642137Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204726.xml
2023-01-11T20:47:31.8642727Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T20:47:31.8643240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:31.8643559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:31.8644068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:31.8644435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:31.8644667Z 
2023-01-11T20:47:31.8644744Z Running tests...
2023-01-11T20:47:31.8645043Z ----------------------------------------------------------------------
2023-01-11T20:47:31.8645395Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:47:31.8645605Z 
2023-01-11T20:47:31.8645803Z ----------------------------------------------------------------------
2023-01-11T20:47:31.8646028Z Ran 1 test in 0.001s
2023-01-11T20:47:31.8646141Z 
2023-01-11T20:47:31.8646214Z OK (skipped=1)
2023-01-11T20:47:31.8646325Z 
2023-01-11T20:47:31.8646412Z Generating XML reports...
2023-01-11T20:47:31.8646917Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204728.xml
2023-01-11T20:47:31.8647494Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T20:47:31.8648109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:31.8648481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:31.8648904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:31.8649253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:31.8649417Z 
2023-01-11T20:47:31.8649491Z Running tests...
2023-01-11T20:47:31.8649799Z ----------------------------------------------------------------------
2023-01-11T20:47:31.8650151Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:47:31.8650365Z 
2023-01-11T20:47:31.8650560Z ----------------------------------------------------------------------
2023-01-11T20:47:31.8653316Z Ran 1 test in 0.001s
2023-01-11T20:47:31.8653559Z 
2023-01-11T20:47:31.8653674Z OK (skipped=1)
2023-01-11T20:47:31.8653941Z 
2023-01-11T20:47:31.8654336Z Generating XML reports...
2023-01-11T20:47:31.8655354Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204729.xml
2023-01-11T20:47:31.8656418Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T20:47:31.8657331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:31.8657943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:31.8658710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:31.8659331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:31.8659637Z 
2023-01-11T20:47:31.8659773Z Running tests...
2023-01-11T20:47:31.8660316Z ----------------------------------------------------------------------
2023-01-11T20:47:31.8660962Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:47:31.8661322Z 
2023-01-11T20:47:31.8661907Z ----------------------------------------------------------------------
2023-01-11T20:47:31.8662315Z Ran 1 test in 0.001s
2023-01-11T20:47:31.8662529Z 
2023-01-11T20:47:31.8662655Z OK (skipped=1)
2023-01-11T20:47:31.8662845Z 
2023-01-11T20:47:31.8662998Z Generating XML reports...
2023-01-11T20:47:31.8663896Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204731.xml
2023-01-11T20:47:31.8664426Z 
2023-01-11T20:47:31.8681897Z ##[endgroup]
2023-01-11T20:47:31.8682998Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_8jiywey0)
2023-01-11T20:47:31.8683598Z 
2023-01-11T20:47:31.8683970Z Running distributed tests for the gloo backend with file init_method in shard 1 of 2
2023-01-11T20:47:31.8685060Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:47:31.856897]
2023-01-11T20:47:46.4383170Z 
2023-01-11T20:47:46.4383824Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T20:47:46.4385293Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_ed8jzyo6)
2023-01-11T20:47:46.4386729Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2023-01-11T20:47:46.4388078Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2023-01-11T20:47:46.4388551Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2023-01-11T20:47:46.4389034Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2023-01-11T20:47:46.4389518Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2023-01-11T20:47:46.4389840Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2023-01-11T20:47:46.4390104Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2023-01-11T20:47:46.4390358Z <unittest.suite.TestSuite tests=[]>
2023-01-11T20:47:46.4390906Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T20:47:46.4391436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:46.4391779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:46.4392476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:46.4393005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:46.4393240Z 
2023-01-11T20:47:46.4393356Z Running tests...
2023-01-11T20:47:46.4393863Z ----------------------------------------------------------------------
2023-01-11T20:47:46.4394582Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3670
2023-01-11T20:47:46.4395011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3671
2023-01-11T20:47:46.4395340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3672
2023-01-11T20:47:46.4395800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:46.4396132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:46.4396547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:46.4396898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:46.4397220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:46.4397682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:46.4398106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:46.4398543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:46.4399015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:46.4399325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:46.4399790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:46.4400116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:46.4400544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:46.4400876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:46.4401197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:46.4401560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:46.4401913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:46.4402433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:46.4402968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T20:47:46.4403497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T20:47:46.4404097Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T20:47:46.4404374Z ok (1.972s)
2023-01-11T20:47:46.4404476Z 
2023-01-11T20:47:46.4404675Z ----------------------------------------------------------------------
2023-01-11T20:47:46.4404916Z Ran 1 test in 1.972s
2023-01-11T20:47:46.4405028Z 
2023-01-11T20:47:46.4405090Z OK
2023-01-11T20:47:46.4405168Z 
2023-01-11T20:47:46.4405252Z Generating XML reports...
2023-01-11T20:47:46.4405766Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204734.xml
2023-01-11T20:47:46.4406356Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T20:47:46.4406849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:46.4407182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:46.4407607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:46.4407949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:46.4408110Z 
2023-01-11T20:47:46.4408169Z Running tests...
2023-01-11T20:47:46.4408469Z ----------------------------------------------------------------------
2023-01-11T20:47:46.4408855Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3723
2023-01-11T20:47:46.4409236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3724
2023-01-11T20:47:46.4409565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3725
2023-01-11T20:47:46.4410015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:46.4410346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:46.4410764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:46.4411112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:46.4411433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:47:46.4411892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:46.4412210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:46.4412641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:46.4412983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:46.4413291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:47:46.4413748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:46.4414073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:46.4414500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:46.4414830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:46.4415154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:47:46.4415578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:47:46.4415965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:47:46.4416334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:47:46.4416832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T20:47:46.4417351Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T20:47:46.4417852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T20:47:46.4418137Z ok (1.980s)
2023-01-11T20:47:46.4418240Z 
2023-01-11T20:47:46.4418441Z ----------------------------------------------------------------------
2023-01-11T20:47:46.4418680Z Ran 1 test in 1.981s
2023-01-11T20:47:46.4418779Z 
2023-01-11T20:47:46.4418841Z OK
2023-01-11T20:47:46.4418932Z 
2023-01-11T20:47:46.4419019Z Generating XML reports...
2023-01-11T20:47:46.4419526Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204737.xml
2023-01-11T20:47:46.4420096Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T20:47:46.4420599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:46.4420928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:46.4421355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:46.4422002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:46.4422193Z 
2023-01-11T20:47:46.4422268Z Running tests...
2023-01-11T20:47:46.4422574Z ----------------------------------------------------------------------
2023-01-11T20:47:46.4422916Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:47:46.4423131Z 
2023-01-11T20:47:46.4423329Z ----------------------------------------------------------------------
2023-01-11T20:47:46.4423568Z Ran 1 test in 0.001s
2023-01-11T20:47:46.4423678Z 
2023-01-11T20:47:46.4423750Z OK (skipped=1)
2023-01-11T20:47:46.4423859Z 
2023-01-11T20:47:46.4423929Z Generating XML reports...
2023-01-11T20:47:46.4424428Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204741.xml
2023-01-11T20:47:46.4425011Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T20:47:46.4425517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:46.4425838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:46.4426266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:46.4426618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:46.4426782Z 
2023-01-11T20:47:46.4426841Z Running tests...
2023-01-11T20:47:46.4427141Z ----------------------------------------------------------------------
2023-01-11T20:47:46.4427492Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:47:46.4427699Z 
2023-01-11T20:47:46.4427893Z ----------------------------------------------------------------------
2023-01-11T20:47:46.4428123Z Ran 1 test in 0.001s
2023-01-11T20:47:46.4428235Z 
2023-01-11T20:47:46.4428414Z OK (skipped=1)
2023-01-11T20:47:46.4428521Z 
2023-01-11T20:47:46.4428608Z Generating XML reports...
2023-01-11T20:47:46.4429143Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204742.xml
2023-01-11T20:47:46.4429730Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T20:47:46.4430237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:46.4430566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:46.4430981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:46.4431323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:46.4431487Z 
2023-01-11T20:47:46.4431559Z Running tests...
2023-01-11T20:47:46.4431860Z ----------------------------------------------------------------------
2023-01-11T20:47:46.4432208Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:47:46.4432423Z 
2023-01-11T20:47:46.4432620Z ----------------------------------------------------------------------
2023-01-11T20:47:46.4432853Z Ran 1 test in 0.001s
2023-01-11T20:47:46.4432963Z 
2023-01-11T20:47:46.4433025Z OK (skipped=1)
2023-01-11T20:47:46.4433132Z 
2023-01-11T20:47:46.4433215Z Generating XML reports...
2023-01-11T20:47:46.4433714Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204744.xml
2023-01-11T20:47:46.4434298Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T20:47:46.4434795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:47:46.4435126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:47:46.4435559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:47:46.4435900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:47:46.4436050Z 
2023-01-11T20:47:46.4436122Z Running tests...
2023-01-11T20:47:46.4436423Z ----------------------------------------------------------------------
2023-01-11T20:47:46.4436781Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:47:46.4436989Z 
2023-01-11T20:47:46.4437184Z ----------------------------------------------------------------------
2023-01-11T20:47:46.4437418Z Ran 1 test in 0.001s
2023-01-11T20:47:46.4437528Z 
2023-01-11T20:47:46.4437601Z OK (skipped=1)
2023-01-11T20:47:46.4437705Z 
2023-01-11T20:47:46.4437788Z Generating XML reports...
2023-01-11T20:47:46.4438275Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204745.xml
2023-01-11T20:47:46.4438571Z 
2023-01-11T20:47:46.4438903Z ##[endgroup]
2023-01-11T20:47:46.4439450Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_ed8jzyo6)
2023-01-11T20:47:46.4439775Z 
2023-01-11T20:47:46.4439964Z Running distributed/test_distributed_spawn ... [2023-01-11 20:47:46.439697]
2023-01-11T20:47:46.4462810Z MPI not available -- MPI backend tests will be skipped
2023-01-11T20:47:46.4463251Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 1}
2023-01-11T20:47:46.4465364Z Running distributed tests for the test backend with env init_method in shard 1 of 2
2023-01-11T20:47:46.4471691Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:47:46.446964]
2023-01-11T20:47:48.1113419Z 
2023-01-11T20:47:48.1114388Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2023-01-11T20:47:48.1115248Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_974cd1jf)
2023-01-11T20:47:48.1115626Z <unittest.suite.TestSuite tests=[]>
2023-01-11T20:47:48.1115770Z 
2023-01-11T20:47:48.1115991Z ##[endgroup]
2023-01-11T20:47:48.1116545Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_974cd1jf)
2023-01-11T20:47:48.1116807Z 
2023-01-11T20:47:48.1123685Z Running distributed tests for the test backend with file init_method in shard 1 of 2
2023-01-11T20:47:48.1128321Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:47:48.112624]
2023-01-11T20:47:49.8065825Z 
2023-01-11T20:47:49.8066385Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2023-01-11T20:47:49.8067610Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_fst0_1p_)
2023-01-11T20:47:49.8067991Z <unittest.suite.TestSuite tests=[]>
2023-01-11T20:47:49.8068120Z 
2023-01-11T20:47:49.8068343Z ##[endgroup]
2023-01-11T20:47:49.8068895Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_fst0_1p_)
2023-01-11T20:47:49.8069159Z 
2023-01-11T20:47:49.8076655Z Running distributed tests for the gloo backend with env init_method in shard 1 of 2
2023-01-11T20:47:49.8081690Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:47:49.807974]
2023-01-11T21:03:04.5586191Z 
2023-01-11T21:03:04.5586769Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2023-01-11T21:03:04.5588322Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_z9rr3hxb)
2023-01-11T21:03:04.5598026Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:03:04.5636997Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_grad_as_bucket_view_false>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_ignored_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2023-01-11T21:03:04.5670478Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5671054Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5671473Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5673675Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5674219Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5674937Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5675525Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5676135Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5676769Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5677452Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5678119Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5678778Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5679444Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5680064Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5680646Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5681245Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5681810Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5682319Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5682893Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5683443Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5684040Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5684758Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5685253Z test_all_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5685820Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5686339Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5686857Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5687373Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5687894Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5688365Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5688855Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5689340Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5689810Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5690294Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5690794Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5691310Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5691792Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5692279Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5692797Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5693276Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5693777Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5694280Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5694827Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5695354Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5695887Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5696429Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5696934Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5697466Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5698002Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5698506Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5699039Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5699580Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5700084Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5700577Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5701091Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5701614Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5702518Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5703000Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5703490Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5703969Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5704418Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5704890Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5705375Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5705823Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5706362Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5706834Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5707382Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5707849Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5708318Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5708774Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5709217Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5709689Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5710173Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5710640Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5711117Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5711594Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5712049Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5712485Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5712948Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5713437Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5713925Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5714386Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5714854Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5715350Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5715861Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5716461Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5717066Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5717612Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5718163Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5718725Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5719267Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5719778Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5720307Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5720843Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5721394Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5721943Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5722498Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5723123Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5723648Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5724175Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5724660Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5725123Z test_backend_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5725549Z test_barrier (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5725977Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5726441Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5726894Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5727449Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5727912Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5728441Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5728918Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5729407Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5729906Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5730379Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5730903Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5731410Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5731917Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5732416Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5784216Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5784927Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5785250Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5785566Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5785860Z test_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5786144Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5786425Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5786719Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5787012Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5787296Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5787638Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5788034Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5788387Z test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5788717Z test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5789075Z test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5789400Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5789695Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5790006Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5790331Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5790669Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5791013Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5791347Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5791665Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5791982Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5792289Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5792565Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5804504Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5804855Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5825766Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5826134Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5826849Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5827350Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5827951Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5828280Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5828634Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5829034Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5829473Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5829908Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5830344Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5830782Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5831213Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5831647Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5832072Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5832453Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5832802Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5833109Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5833379Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5833655Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5833936Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5834209Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5834595Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5834905Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5835260Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5835630Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5835923Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5836192Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5836478Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5836781Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5837079Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5837367Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5837645Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5837942Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5838241Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5838526Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5838807Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5839138Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5839480Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5839763Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5840071Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5840419Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5840726Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5841019Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5841301Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5841580Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5841892Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5842206Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5842491Z test_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5842806Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5843084Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5843365Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5843630Z test_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5843903Z test_gather_object (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5844188Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5844456Z test_get_backend (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5844726Z test_get_future (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5844990Z test_get_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5845270Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5845550Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5845839Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5846113Z test_irecv (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5846353Z test_isend (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5846632Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5846923Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5847213Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5847548Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5847883Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5848188Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5848482Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5848810Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5849133Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5849432Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5849741Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5850043Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5850341Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5850632Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5850920Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5851213Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5851546Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5851949Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5852326Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5852643Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5852984Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5853320Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5853637Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5853934Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5854250Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5854575Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5854894Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5855251Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5855631Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5855984Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5856280Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5856567Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5856868Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5857149Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5857431Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5857707Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5857990Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5858261Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5858533Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5858803Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5859062Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5859339Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5859629Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5859913Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5860185Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5860456Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5860740Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5861008Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5861279Z test_scatter (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5861547Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5861980Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5862257Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5862540Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5862817Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5863099Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5863379Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5863639Z test_send_recv (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5863916Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5864224Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5864547Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5864928Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5865219Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5865554Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5865858Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5866166Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5866455Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5866754Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5867062Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5867365Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5867657Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5867940Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5868235Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5868518Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5868812Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5869146Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5869468Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.5869975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.5870447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5870786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5871218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5871558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5871725Z 
2023-01-11T21:03:04.5871800Z Running tests...
2023-01-11T21:03:04.5872112Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5872573Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3847
2023-01-11T21:03:04.5873010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3848
2023-01-11T21:03:04.5873340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3849
2023-01-11T21:03:04.5873793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5874122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5874540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5874885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5875221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.5875674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5875996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5876421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5876766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5877083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.5877551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5877917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5878382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5878713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5879049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.5879544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5880050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5880563Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5880956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.5881307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.5881641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.5881926Z skip: Need at least 2 CUDA devices (2.174s)
2023-01-11T21:03:04.5882063Z 
2023-01-11T21:03:04.5882264Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5882508Z Ran 1 test in 2.174s
2023-01-11T21:03:04.5882608Z 
2023-01-11T21:03:04.5882778Z OK (skipped=1)
2023-01-11T21:03:04.5882890Z 
2023-01-11T21:03:04.5882976Z Generating XML reports...
2023-01-11T21:03:04.5883443Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204752.xml
2023-01-11T21:03:04.5883930Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.5884397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5884733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5885164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5885499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5885665Z 
2023-01-11T21:03:04.5885740Z Running tests...
2023-01-11T21:03:04.5886047Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5886405Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2023-01-11T21:03:04.5886633Z 
2023-01-11T21:03:04.5886827Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5887061Z Ran 1 test in 0.003s
2023-01-11T21:03:04.5887233Z 
2023-01-11T21:03:04.5887303Z OK (skipped=1)
2023-01-11T21:03:04.5887397Z 
2023-01-11T21:03:04.5887481Z Generating XML reports...
2023-01-11T21:03:04.5887946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204756.xml
2023-01-11T21:03:04.5888436Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.5888881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5889209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5889634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5889982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5890142Z 
2023-01-11T21:03:04.5890201Z Running tests...
2023-01-11T21:03:04.5890548Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5890974Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3910
2023-01-11T21:03:04.5891370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3911
2023-01-11T21:03:04.5891683Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3912
2023-01-11T21:03:04.5892132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5892459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5892872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5893225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5893562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.5894032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5894345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5894773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5895114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5895435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.5895900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5896223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5896644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5896974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5897309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.5897807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5898327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5898826Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5899210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.5899551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.5899880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.5900132Z ok (2.179s)
2023-01-11T21:03:04.5900235Z 
2023-01-11T21:03:04.5900439Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5900676Z Ran 1 test in 2.179s
2023-01-11T21:03:04.5900777Z 
2023-01-11T21:03:04.5900840Z OK
2023-01-11T21:03:04.5900933Z 
2023-01-11T21:03:04.5901016Z Generating XML reports...
2023-01-11T21:03:04.5901478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204758.xml
2023-01-11T21:03:04.5902076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.5902547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5902877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5903304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5903701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5903865Z 
2023-01-11T21:03:04.5903975Z Running tests...
2023-01-11T21:03:04.5904287Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5905178Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.258s)
2023-01-11T21:03:04.5905732Z 
2023-01-11T21:03:04.5905930Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5906156Z Ran 1 test in 0.258s
2023-01-11T21:03:04.5906272Z 
2023-01-11T21:03:04.5906346Z OK (skipped=1)
2023-01-11T21:03:04.5906458Z 
2023-01-11T21:03:04.5906544Z Generating XML reports...
2023-01-11T21:03:04.5906990Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204802.xml
2023-01-11T21:03:04.5907493Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.5907959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5908288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5908701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5909045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5909206Z 
2023-01-11T21:03:04.5909277Z Running tests...
2023-01-11T21:03:04.5909566Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5909987Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3973
2023-01-11T21:03:04.5910410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3974
2023-01-11T21:03:04.5910740Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3975
2023-01-11T21:03:04.5911179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5911507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5911933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5912275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5912600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.5913068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5913395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5913814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5914159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5914489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.5914957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5915269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5915700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5916040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5916421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.5916927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5917447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5917963Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5918332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.5918686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.5919028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.5919387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.5919729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.5920087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.5920339Z ok (2.279s)
2023-01-11T21:03:04.5920444Z 
2023-01-11T21:03:04.5920631Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5920869Z Ran 1 test in 2.279s
2023-01-11T21:03:04.5920978Z 
2023-01-11T21:03:04.5921040Z OK
2023-01-11T21:03:04.5921130Z 
2023-01-11T21:03:04.5921213Z Generating XML reports...
2023-01-11T21:03:04.5921657Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204804.xml
2023-01-11T21:03:04.5922153Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.5922615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5923012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5923445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5923792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5923956Z 
2023-01-11T21:03:04.5924032Z Running tests...
2023-01-11T21:03:04.5924323Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5924763Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4026
2023-01-11T21:03:04.5925194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4027
2023-01-11T21:03:04.5925526Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4028
2023-01-11T21:03:04.5925962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5926289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5926718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5927054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5927389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.5927861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5928186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5928600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5928989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5929324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.5929835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5930156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5930589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5930930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5931246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.5931744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5932263Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5932789Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5933161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.5933511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.5933859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.5934209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.5934551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.5934907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.5935162Z ok (2.285s)
2023-01-11T21:03:04.5935268Z 
2023-01-11T21:03:04.5935458Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5935696Z Ran 1 test in 2.285s
2023-01-11T21:03:04.5935810Z 
2023-01-11T21:03:04.5935874Z OK
2023-01-11T21:03:04.5935968Z 
2023-01-11T21:03:04.5936038Z Generating XML reports...
2023-01-11T21:03:04.5936496Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204808.xml
2023-01-11T21:03:04.5936993Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.5937454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5937768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5938196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5938540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5938705Z 
2023-01-11T21:03:04.5938775Z Running tests...
2023-01-11T21:03:04.5939062Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5939492Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4079
2023-01-11T21:03:04.5939926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4080
2023-01-11T21:03:04.5940242Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4081
2023-01-11T21:03:04.5940693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5941022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5941449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5941950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5942288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.5942818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5943134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5943565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5943917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5944256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.5944710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5945046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5945477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5945825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5946146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.5946640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5947160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5947662Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5948041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.5948388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.5948732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.5948996Z skip: CUDA is not available. (2.174s)
2023-01-11T21:03:04.5949125Z 
2023-01-11T21:03:04.5949324Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5949564Z Ran 1 test in 2.175s
2023-01-11T21:03:04.5949676Z 
2023-01-11T21:03:04.5949752Z OK (skipped=1)
2023-01-11T21:03:04.5949846Z 
2023-01-11T21:03:04.5949933Z Generating XML reports...
2023-01-11T21:03:04.5950390Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204812.xml
2023-01-11T21:03:04.5950880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.5951324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5951654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5952084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5952425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5952574Z 
2023-01-11T21:03:04.5952648Z Running tests...
2023-01-11T21:03:04.5952946Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5953382Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4132
2023-01-11T21:03:04.5953804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4133
2023-01-11T21:03:04.5954129Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4134
2023-01-11T21:03:04.5954577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5954952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5955399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5955740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5956073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.5956541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5956852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5957275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5957616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5957936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.5958405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5958732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5959152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5959480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5959818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.5960316Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5960831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5961338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5961732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.5962079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.5962407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.5962756Z skip: CUDA is not available. (2.181s)
2023-01-11T21:03:04.5962893Z 
2023-01-11T21:03:04.5963095Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5963336Z Ran 1 test in 2.181s
2023-01-11T21:03:04.5963434Z 
2023-01-11T21:03:04.5963508Z OK (skipped=1)
2023-01-11T21:03:04.5963616Z 
2023-01-11T21:03:04.5963701Z Generating XML reports...
2023-01-11T21:03:04.5964161Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204816.xml
2023-01-11T21:03:04.5964648Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.5965113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5965445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5965876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5966206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5966365Z 
2023-01-11T21:03:04.5966437Z Running tests...
2023-01-11T21:03:04.5966738Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5967168Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4185
2023-01-11T21:03:04.5967654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4186
2023-01-11T21:03:04.5968012Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4187
2023-01-11T21:03:04.5968464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5968779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5969208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5969552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5969890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.5970346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5970680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5971108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5971436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5971772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.5972238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5972563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5972977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5973317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5973650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.5974147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5974653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5975166Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5975550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.5975884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.5976230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.5976502Z skip: CUDA is not available. (2.176s)
2023-01-11T21:03:04.5976633Z 
2023-01-11T21:03:04.5976837Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5977061Z Ran 1 test in 2.176s
2023-01-11T21:03:04.5977172Z 
2023-01-11T21:03:04.5977247Z OK (skipped=1)
2023-01-11T21:03:04.5977354Z 
2023-01-11T21:03:04.5977440Z Generating XML reports...
2023-01-11T21:03:04.5977883Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204820.xml
2023-01-11T21:03:04.5978379Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.5978841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5979172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5979586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5979929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5980127Z 
2023-01-11T21:03:04.5980200Z Running tests...
2023-01-11T21:03:04.5980500Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5980977Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4238
2023-01-11T21:03:04.5981435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4239
2023-01-11T21:03:04.5981882Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4240
2023-01-11T21:03:04.5982328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5982660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5983089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5983444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5983769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.5984240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5984572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5985001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5985329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5985667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.5986132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5986442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5986871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5987219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5987551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.5988027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5988543Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5989057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.5989438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.5989773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.5990115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.5990392Z skip: CUDA is not available. (2.176s)
2023-01-11T21:03:04.5990527Z 
2023-01-11T21:03:04.5990713Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5990951Z Ran 1 test in 2.176s
2023-01-11T21:03:04.5991067Z 
2023-01-11T21:03:04.5991139Z OK (skipped=1)
2023-01-11T21:03:04.5991249Z 
2023-01-11T21:03:04.5991331Z Generating XML reports...
2023-01-11T21:03:04.5991776Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204824.xml
2023-01-11T21:03:04.5992270Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.5992732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5993102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5993563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5993917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5994084Z 
2023-01-11T21:03:04.5994158Z Running tests...
2023-01-11T21:03:04.5994456Z ----------------------------------------------------------------------
2023-01-11T21:03:04.5994915Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4291
2023-01-11T21:03:04.5995366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4292
2023-01-11T21:03:04.5995681Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4293
2023-01-11T21:03:04.5996136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5996482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5996917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5997251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5997584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.5998057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.5998389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.5998803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.5999146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.5999483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.5999953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6000279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6000709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6001048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6001372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6001863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6002379Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6002963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6003340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6003692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6004042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6004306Z skip: CUDA is not available. (2.177s)
2023-01-11T21:03:04.6004436Z 
2023-01-11T21:03:04.6004637Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6004880Z Ran 1 test in 2.177s
2023-01-11T21:03:04.6004995Z 
2023-01-11T21:03:04.6005069Z OK (skipped=1)
2023-01-11T21:03:04.6005164Z 
2023-01-11T21:03:04.6005249Z Generating XML reports...
2023-01-11T21:03:04.6005714Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204828.xml
2023-01-11T21:03:04.6006284Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6006741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6007074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6007502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6007849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6008010Z 
2023-01-11T21:03:04.6008070Z Running tests...
2023-01-11T21:03:04.6008372Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6008812Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4344
2023-01-11T21:03:04.6009252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4345
2023-01-11T21:03:04.6009570Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4346
2023-01-11T21:03:04.6010020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6010351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6010763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6011105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6011437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6011932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6012414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6012750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6013176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6013522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6013844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6014309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6014636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6015051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6015398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6015733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6016223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6016725Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6017107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6017453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6017802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6018066Z skip: CUDA is not available. (2.181s)
2023-01-11T21:03:04.6018195Z 
2023-01-11T21:03:04.6018436Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6018675Z Ran 1 test in 2.182s
2023-01-11T21:03:04.6018787Z 
2023-01-11T21:03:04.6018846Z OK (skipped=1)
2023-01-11T21:03:04.6018999Z 
2023-01-11T21:03:04.6019089Z Generating XML reports...
2023-01-11T21:03:04.6019547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204832.xml
2023-01-11T21:03:04.6020033Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6020491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6020820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6021251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6021585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6021861Z 
2023-01-11T21:03:04.6021936Z Running tests...
2023-01-11T21:03:04.6022240Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6022704Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4397
2023-01-11T21:03:04.6023143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4398
2023-01-11T21:03:04.6023477Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4399
2023-01-11T21:03:04.6023933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6024251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6024685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6025037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6025377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6025831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6026160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6026589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6026931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6027251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6027720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6028045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6028454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6028799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6029136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6029627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6030132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6030651Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6031035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6031440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6031814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6032094Z skip: CUDA is not available. (2.182s)
2023-01-11T21:03:04.6032229Z 
2023-01-11T21:03:04.6032431Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6032660Z Ran 1 test in 2.182s
2023-01-11T21:03:04.6032772Z 
2023-01-11T21:03:04.6032846Z OK (skipped=1)
2023-01-11T21:03:04.6032954Z 
2023-01-11T21:03:04.6033040Z Generating XML reports...
2023-01-11T21:03:04.6033500Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204836.xml
2023-01-11T21:03:04.6033974Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6034435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6034772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6035193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6035538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6035700Z 
2023-01-11T21:03:04.6035773Z Running tests...
2023-01-11T21:03:04.6036073Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6036969Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.263s)
2023-01-11T21:03:04.6037461Z 
2023-01-11T21:03:04.6037661Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6037906Z Ran 1 test in 0.263s
2023-01-11T21:03:04.6038019Z 
2023-01-11T21:03:04.6038090Z OK (skipped=1)
2023-01-11T21:03:04.6038198Z 
2023-01-11T21:03:04.6038270Z Generating XML reports...
2023-01-11T21:03:04.6038729Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204840.xml
2023-01-11T21:03:04.6039222Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6039685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6040004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6040432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6040777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6040942Z 
2023-01-11T21:03:04.6041002Z Running tests...
2023-01-11T21:03:04.6041302Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6041731Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4460
2023-01-11T21:03:04.6042158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4461
2023-01-11T21:03:04.6042472Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4462
2023-01-11T21:03:04.6042979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6043314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6043730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6044121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6044462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6044966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6045287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6045716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6046061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6046396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6046849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6047178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6047605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6047933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6048263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6048757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6049276Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6049772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6050156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6050507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6050853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6051092Z ok (2.278s)
2023-01-11T21:03:04.6051194Z 
2023-01-11T21:03:04.6051397Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6051636Z Ran 1 test in 2.279s
2023-01-11T21:03:04.6051747Z 
2023-01-11T21:03:04.6051797Z OK
2023-01-11T21:03:04.6051886Z 
2023-01-11T21:03:04.6051971Z Generating XML reports...
2023-01-11T21:03:04.6052430Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204842.xml
2023-01-11T21:03:04.6052919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6053366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6053700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6054132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6054466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6054632Z 
2023-01-11T21:03:04.6054705Z Running tests...
2023-01-11T21:03:04.6055003Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6055907Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.262s)
2023-01-11T21:03:04.6056401Z 
2023-01-11T21:03:04.6056595Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6056858Z Ran 1 test in 0.262s
2023-01-11T21:03:04.6056969Z 
2023-01-11T21:03:04.6057042Z OK (skipped=1)
2023-01-11T21:03:04.6057149Z 
2023-01-11T21:03:04.6057272Z Generating XML reports...
2023-01-11T21:03:04.6057721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204846.xml
2023-01-11T21:03:04.6058216Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6058680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6059010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6059420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6059765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6059929Z 
2023-01-11T21:03:04.6060000Z Running tests...
2023-01-11T21:03:04.6060287Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6060701Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4523
2023-01-11T21:03:04.6061118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4524
2023-01-11T21:03:04.6061453Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4525
2023-01-11T21:03:04.6062002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6062336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6062768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6063117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6063479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6063958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6064289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6064702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6065051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6065379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6065845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6066161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6066591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6066938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6067260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6067752Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6068269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6068776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6069144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6069493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6069903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6070228Z skip: Need at least 3 CUDA devices (2.278s)
2023-01-11T21:03:04.6070352Z 
2023-01-11T21:03:04.6070554Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6070794Z Ran 1 test in 2.278s
2023-01-11T21:03:04.6070906Z 
2023-01-11T21:03:04.6070978Z OK (skipped=1)
2023-01-11T21:03:04.6071089Z 
2023-01-11T21:03:04.6071160Z Generating XML reports...
2023-01-11T21:03:04.6071625Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204848.xml
2023-01-11T21:03:04.6072119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6072580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6072895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6073323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6073668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6073831Z 
2023-01-11T21:03:04.6073890Z Running tests...
2023-01-11T21:03:04.6074188Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6074593Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.001s)
2023-01-11T21:03:04.6074802Z 
2023-01-11T21:03:04.6074998Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6075224Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6075334Z 
2023-01-11T21:03:04.6075406Z OK (skipped=1)
2023-01-11T21:03:04.6075513Z 
2023-01-11T21:03:04.6075599Z Generating XML reports...
2023-01-11T21:03:04.6076040Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204852.xml
2023-01-11T21:03:04.6076538Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6077004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6077324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6077755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6078103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6078253Z 
2023-01-11T21:03:04.6078328Z Running tests...
2023-01-11T21:03:04.6078629Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6078944Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.6079331Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4586
2023-01-11T21:03:04.6079726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4587
2023-01-11T21:03:04.6080055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4588
2023-01-11T21:03:04.6080505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6080824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6081252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6081601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6081925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6082400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6082846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6083311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6083647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6083982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6084453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6084782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6085198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6085541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6085878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6086362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6086886Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6087396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6087781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6088111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6088451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6088697Z ok (2.176s)
2023-01-11T21:03:04.6088801Z 
2023-01-11T21:03:04.6088999Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6089221Z Ran 1 test in 2.176s
2023-01-11T21:03:04.6089330Z 
2023-01-11T21:03:04.6089391Z OK
2023-01-11T21:03:04.6089484Z 
2023-01-11T21:03:04.6089567Z Generating XML reports...
2023-01-11T21:03:04.6090011Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204854.xml
2023-01-11T21:03:04.6090501Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6090958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6091285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6091696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6092037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6092200Z 
2023-01-11T21:03:04.6092271Z Running tests...
2023-01-11T21:03:04.6092560Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6092893Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.6093303Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4639
2023-01-11T21:03:04.6093697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4640
2023-01-11T21:03:04.6094011Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4641
2023-01-11T21:03:04.6094456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6094782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6095199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6095578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6095940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6096409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6096718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6097143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6097484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6097814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6098268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6098595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6099025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6099352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6099678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6100167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6100682Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6101179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6101565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6102029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6102376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6102713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.6103069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.6103423Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.6103665Z ok (2.180s)
2023-01-11T21:03:04.6103768Z 
2023-01-11T21:03:04.6103969Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6104210Z Ran 1 test in 2.180s
2023-01-11T21:03:04.6104323Z 
2023-01-11T21:03:04.6104383Z OK
2023-01-11T21:03:04.6104461Z 
2023-01-11T21:03:04.6104543Z Generating XML reports...
2023-01-11T21:03:04.6105003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204858.xml
2023-01-11T21:03:04.6105493Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6105941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6106269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6106695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6107039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6107190Z 
2023-01-11T21:03:04.6107259Z Running tests...
2023-01-11T21:03:04.6107559Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6107899Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.6108436Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4692
2023-01-11T21:03:04.6108848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4693
2023-01-11T21:03:04.6109177Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4694
2023-01-11T21:03:04.6109630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6109945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6110369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6110711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6111045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6111507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6111838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6112264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6112591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6112921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6113387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6113709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6114118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6114458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6114795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6115276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6115793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6116309Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6116690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6117020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6117359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6117705Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.6118062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.6118399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.6118653Z ok (2.182s)
2023-01-11T21:03:04.6118754Z 
2023-01-11T21:03:04.6118952Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6119175Z Ran 1 test in 2.183s
2023-01-11T21:03:04.6119286Z 
2023-01-11T21:03:04.6119347Z OK
2023-01-11T21:03:04.6119438Z 
2023-01-11T21:03:04.6119521Z Generating XML reports...
2023-01-11T21:03:04.6119974Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204902.xml
2023-01-11T21:03:04.6120454Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6120951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6121307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6121721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6122068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6122230Z 
2023-01-11T21:03:04.6122302Z Running tests...
2023-01-11T21:03:04.6122595Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6122987Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:03:04.6123395Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4745
2023-01-11T21:03:04.6123789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4746
2023-01-11T21:03:04.6124104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4747
2023-01-11T21:03:04.6124560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6124892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6125320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6125653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6125992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6126458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6126783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6127200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6127544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6127875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6128326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6128648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6129073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6129414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6129729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6130225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6130744Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6131257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6131624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6131971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6132314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6132551Z ok (2.278s)
2023-01-11T21:03:04.6132651Z 
2023-01-11T21:03:04.6132851Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6133091Z Ran 1 test in 2.278s
2023-01-11T21:03:04.6133245Z 
2023-01-11T21:03:04.6133305Z OK
2023-01-11T21:03:04.6133382Z 
2023-01-11T21:03:04.6133468Z Generating XML reports...
2023-01-11T21:03:04.6133961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204906.xml
2023-01-11T21:03:04.6134461Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6134913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6135247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6135677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6136024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6136176Z 
2023-01-11T21:03:04.6136253Z Running tests...
2023-01-11T21:03:04.6136557Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6136952Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4798
2023-01-11T21:03:04.6137332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4799
2023-01-11T21:03:04.6137662Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4800
2023-01-11T21:03:04.6138112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6138445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6138866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6139215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6139553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6140033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6140350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6140778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6141119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6141443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6142059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6142583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6143088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6143408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6143844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6144189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6144524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6145001Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6145169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6145336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6145494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6145805Z STAGE:2023-01-11 20:49:12 4799:4799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6146252Z STAGE:2023-01-11 20:49:12 4800:4800 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:49:12 4798:4798 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6146260Z 
2023-01-11T21:03:04.6146514Z STAGE:2023-01-11 20:49:12 4800:4800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6146763Z STAGE:2023-01-11 20:49:12 4798:4798 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6146994Z STAGE:2023-01-11 20:49:12 4799:4799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6147249Z STAGE:2023-01-11 20:49:12 4800:4800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6147501Z STAGE:2023-01-11 20:49:12 4798:4798 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6147760Z STAGE:2023-01-11 20:49:12 4799:4799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6148000Z STAGE:2023-01-11 20:49:12 4798:4798 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6148393Z STAGE:2023-01-11 20:49:12 4800:4800 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:49:12 4799:4799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6148399Z 
2023-01-11T21:03:04.6148642Z STAGE:2023-01-11 20:49:12 4800:4800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6148893Z STAGE:2023-01-11 20:49:12 4800:4800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6149301Z STAGE:2023-01-11 20:49:12 4798:4798 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:12 4799:4799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6149310Z 
2023-01-11T21:03:04.6149732Z STAGE:2023-01-11 20:49:12 4798:4798 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:12 4799:4799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6149738Z 
2023-01-11T21:03:04.6149976Z STAGE:2023-01-11 20:49:12 4800:4800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6150356Z STAGE:2023-01-11 20:49:12 4798:4798 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:49:12 4799:4799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6150373Z 
2023-01-11T21:03:04.6150602Z STAGE:2023-01-11 20:49:12 4800:4800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6150854Z STAGE:2023-01-11 20:49:12 4800:4800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6151258Z STAGE:2023-01-11 20:49:12 4798:4798 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:12 4799:4799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6151266Z 
2023-01-11T21:03:04.6151687Z STAGE:2023-01-11 20:49:12 4798:4798 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:12 4799:4799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6151693Z 
2023-01-11T21:03:04.6151760Z ok (2.277s)
2023-01-11T21:03:04.6151765Z 
2023-01-11T21:03:04.6151965Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6152038Z Ran 1 test in 2.277s
2023-01-11T21:03:04.6152044Z 
2023-01-11T21:03:04.6152105Z OK
2023-01-11T21:03:04.6152110Z 
2023-01-11T21:03:04.6152193Z Generating XML reports...
2023-01-11T21:03:04.6152582Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204910.xml
2023-01-11T21:03:04.6152920Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6153432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6153605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6154019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6154205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6154213Z 
2023-01-11T21:03:04.6154311Z Running tests...
2023-01-11T21:03:04.6154585Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6154922Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4854
2023-01-11T21:03:04.6155154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4855
2023-01-11T21:03:04.6155387Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4856
2023-01-11T21:03:04.6155846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6156041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6156498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6156710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6157019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6157485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6157696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6158190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6158419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6158723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6159231Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6159703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6159918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6160418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6160649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6160929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6161401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6161863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6162030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6162283Z STAGE:2023-01-11 20:49:16 4854:4854 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6162446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6162605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6162920Z STAGE:2023-01-11 20:49:16 4856:4856 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6163161Z STAGE:2023-01-11 20:49:16 4855:4855 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6163525Z STAGE:2023-01-11 20:49:16 4856:4856 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6163789Z STAGE:2023-01-11 20:49:16 4855:4855 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6164048Z STAGE:2023-01-11 20:49:16 4856:4856 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6164294Z STAGE:2023-01-11 20:49:16 4854:4854 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6164546Z STAGE:2023-01-11 20:49:16 4855:4855 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6164795Z STAGE:2023-01-11 20:49:16 4854:4854 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6165190Z STAGE:2023-01-11 20:49:16 4856:4856 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:49:16 4855:4855 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6165202Z 
2023-01-11T21:03:04.6165441Z STAGE:2023-01-11 20:49:16 4854:4854 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6166008Z STAGE:2023-01-11 20:49:16 4855:4855 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:16 4856:4856 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:16 4854:4854 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6166015Z 
2023-01-11T21:03:04.6166019Z 
2023-01-11T21:03:04.6166442Z STAGE:2023-01-11 20:49:16 4855:4855 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:16 4856:4856 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6166448Z 
2023-01-11T21:03:04.6166703Z STAGE:2023-01-11 20:49:16 4854:4854 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6166942Z STAGE:2023-01-11 20:49:16 4855:4855 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6167182Z STAGE:2023-01-11 20:49:16 4856:4856 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6167404Z STAGE:2023-01-11 20:49:16 4854:4854 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6167695Z STAGE:2023-01-11 20:49:16 4856:4856 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6168085Z STAGE:2023-01-11 20:49:16 4854:4854 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6168642Z STAGE:2023-01-11 20:49:16 4856:4856 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:16 4855:4855 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6168648Z 
2023-01-11T21:03:04.6168904Z STAGE:2023-01-11 20:49:16 4854:4854 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6169161Z STAGE:2023-01-11 20:49:16 4855:4855 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6169742Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6169896Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6170448Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6170598Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6171204Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6171376Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6171432Z ok (2.278s)
2023-01-11T21:03:04.6171437Z 
2023-01-11T21:03:04.6171634Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6171711Z Ran 1 test in 2.279s
2023-01-11T21:03:04.6171717Z 
2023-01-11T21:03:04.6171778Z OK
2023-01-11T21:03:04.6171783Z 
2023-01-11T21:03:04.6171866Z Generating XML reports...
2023-01-11T21:03:04.6172210Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204914.xml
2023-01-11T21:03:04.6172441Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6172719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6172834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6173121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6173256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6173262Z 
2023-01-11T21:03:04.6173335Z Running tests...
2023-01-11T21:03:04.6173531Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6173779Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4910
2023-01-11T21:03:04.6173933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4911
2023-01-11T21:03:04.6174086Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4912
2023-01-11T21:03:04.6174362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6174474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6174755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6174889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6175064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6175339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6175461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6175741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6175877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6176043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6176313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6176433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6176712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6176844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6177019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6177316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6177643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6177980Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6178133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6178297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6178456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6178630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6178803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6178972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6179274Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6179570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6179862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6180094Z STAGE:2023-01-11 20:49:20 4912:4912 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6180336Z STAGE:2023-01-11 20:49:20 4911:4911 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6180574Z STAGE:2023-01-11 20:49:20 4910:4910 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6180816Z STAGE:2023-01-11 20:49:20 4911:4911 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6181224Z STAGE:2023-01-11 20:49:20 4912:4912 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:20 4910:4910 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6181232Z 
2023-01-11T21:03:04.6181489Z STAGE:2023-01-11 20:49:20 4911:4911 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6182065Z STAGE:2023-01-11 20:49:20 4912:4912 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:20 4910:4910 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6182073Z 
2023-01-11T21:03:04.6182317Z STAGE:2023-01-11 20:49:20 4911:4911 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6182710Z STAGE:2023-01-11 20:49:20 4910:4910 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:49:20 4912:4912 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6182715Z 
2023-01-11T21:03:04.6183123Z STAGE:2023-01-11 20:49:20 4912:4912 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:20 4910:4910 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6183131Z 
2023-01-11T21:03:04.6183375Z STAGE:2023-01-11 20:49:20 4911:4911 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6183630Z STAGE:2023-01-11 20:49:20 4910:4910 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6183870Z STAGE:2023-01-11 20:49:20 4912:4912 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6184125Z STAGE:2023-01-11 20:49:20 4911:4911 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6184362Z STAGE:2023-01-11 20:49:20 4910:4910 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6184601Z STAGE:2023-01-11 20:49:20 4911:4911 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6184899Z STAGE:2023-01-11 20:49:20 4912:4912 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6185181Z STAGE:2023-01-11 20:49:20 4912:4912 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6185586Z STAGE:2023-01-11 20:49:20 4910:4910 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:20 4911:4911 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6185593Z 
2023-01-11T21:03:04.6185847Z STAGE:2023-01-11 20:49:20 4912:4912 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6186268Z STAGE:2023-01-11 20:49:20 4910:4910 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:20 4911:4911 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6186274Z 
2023-01-11T21:03:04.6186841Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6186998Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6187553Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6187690Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6188248Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6188397Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6188470Z ok (2.284s)
2023-01-11T21:03:04.6188476Z 
2023-01-11T21:03:04.6188682Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6188758Z Ran 1 test in 2.284s
2023-01-11T21:03:04.6188764Z 
2023-01-11T21:03:04.6188824Z OK
2023-01-11T21:03:04.6188829Z 
2023-01-11T21:03:04.6188912Z Generating XML reports...
2023-01-11T21:03:04.6189254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204919.xml
2023-01-11T21:03:04.6189473Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6189746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6189870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6190154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6190294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6190299Z 
2023-01-11T21:03:04.6190374Z Running tests...
2023-01-11T21:03:04.6190572Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6190816Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4975
2023-01-11T21:03:04.6190969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4976
2023-01-11T21:03:04.6191110Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4977
2023-01-11T21:03:04.6191380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6191502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6191817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6191978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6192155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6192429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6192550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6192831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6192953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6193130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6193405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6193531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6193807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6193939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6194116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6194414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6194700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6194998Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6195164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6195330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6195490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6195662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6195836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6196004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6196297Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6196530Z STAGE:2023-01-11 20:49:25 4977:4977 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6196826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6197122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6197369Z STAGE:2023-01-11 20:49:25 4976:4976 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6197614Z STAGE:2023-01-11 20:49:25 4976:4976 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6197856Z STAGE:2023-01-11 20:49:25 4977:4977 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6198108Z STAGE:2023-01-11 20:49:25 4976:4976 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6198357Z STAGE:2023-01-11 20:49:25 4977:4977 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6198627Z STAGE:2023-01-11 20:49:25 4976:4976 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6198879Z STAGE:2023-01-11 20:49:25 4977:4977 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6199288Z STAGE:2023-01-11 20:49:25 4977:4977 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:25 4976:4976 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6199295Z 
2023-01-11T21:03:04.6199715Z STAGE:2023-01-11 20:49:25 4976:4976 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:25 4977:4977 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6199720Z 
2023-01-11T21:03:04.6199958Z STAGE:2023-01-11 20:49:25 4976:4976 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6200193Z STAGE:2023-01-11 20:49:25 4977:4977 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6200436Z STAGE:2023-01-11 20:49:25 4977:4977 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6200681Z STAGE:2023-01-11 20:49:25 4976:4976 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6200937Z STAGE:2023-01-11 20:49:25 4977:4977 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6201187Z STAGE:2023-01-11 20:49:25 4976:4976 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6201754Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6201905Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6202459Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6202597Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6202740Z ok (2.277s)
2023-01-11T21:03:04.6202746Z 
2023-01-11T21:03:04.6202948Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6203025Z Ran 1 test in 2.278s
2023-01-11T21:03:04.6203031Z 
2023-01-11T21:03:04.6203094Z OK
2023-01-11T21:03:04.6203100Z 
2023-01-11T21:03:04.6203184Z Generating XML reports...
2023-01-11T21:03:04.6203526Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204923.xml
2023-01-11T21:03:04.6203761Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6204039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6204154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6204438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6204574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6204580Z 
2023-01-11T21:03:04.6204654Z Running tests...
2023-01-11T21:03:04.6204853Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6205099Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5036
2023-01-11T21:03:04.6205257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5037
2023-01-11T21:03:04.6205410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5038
2023-01-11T21:03:04.6205708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6205860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6206143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6206278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6206456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6206728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6206854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6207135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6207272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6207436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6207712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6207835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6208118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6208252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6208430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6208732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6209029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6209327Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6209482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6209648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6209809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6210213Z STAGE:2023-01-11 20:49:29 5037:5037 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:49:29 5036:5036 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6210219Z 
2023-01-11T21:03:04.6210461Z STAGE:2023-01-11 20:49:29 5038:5038 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6210708Z STAGE:2023-01-11 20:49:29 5036:5036 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6211120Z STAGE:2023-01-11 20:49:29 5038:5038 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:29 5037:5037 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6211126Z 
2023-01-11T21:03:04.6211380Z STAGE:2023-01-11 20:49:29 5036:5036 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6211804Z STAGE:2023-01-11 20:49:29 5037:5037 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:29 5038:5038 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6211810Z 
2023-01-11T21:03:04.6212199Z STAGE:2023-01-11 20:49:29 5036:5036 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:49:29 5037:5037 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6212439Z STAGE:2023-01-11 20:49:29 5038:5038 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6212473Z 
2023-01-11T21:03:04.6212901Z STAGE:2023-01-11 20:49:29 5036:5036 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:29 5038:5038 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6212908Z 
2023-01-11T21:03:04.6213154Z STAGE:2023-01-11 20:49:29 5037:5037 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6213564Z STAGE:2023-01-11 20:49:29 5036:5036 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:29 5038:5038 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6213581Z 
2023-01-11T21:03:04.6213821Z STAGE:2023-01-11 20:49:29 5037:5037 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6214209Z STAGE:2023-01-11 20:49:29 5036:5036 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:49:29 5038:5038 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6214217Z 
2023-01-11T21:03:04.6214454Z STAGE:2023-01-11 20:49:29 5037:5037 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6214698Z STAGE:2023-01-11 20:49:29 5036:5036 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6215113Z STAGE:2023-01-11 20:49:29 5036:5036 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:29 5037:5037 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6215119Z 
2023-01-11T21:03:04.6215368Z STAGE:2023-01-11 20:49:29 5038:5038 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6215624Z STAGE:2023-01-11 20:49:29 5037:5037 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6215876Z STAGE:2023-01-11 20:49:29 5038:5038 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6216452Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6216603Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6217166Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6217302Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6217851Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6218002Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6218071Z ok (2.284s)
2023-01-11T21:03:04.6218077Z 
2023-01-11T21:03:04.6218279Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6218353Z Ran 1 test in 2.284s
2023-01-11T21:03:04.6218359Z 
2023-01-11T21:03:04.6218419Z OK
2023-01-11T21:03:04.6218424Z 
2023-01-11T21:03:04.6218507Z Generating XML reports...
2023-01-11T21:03:04.6218848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204927.xml
2023-01-11T21:03:04.6219066Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6219339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6219496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6219803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6219938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6219943Z 
2023-01-11T21:03:04.6220017Z Running tests...
2023-01-11T21:03:04.6220211Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6220457Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5092
2023-01-11T21:03:04.6220610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5093
2023-01-11T21:03:04.6220750Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5094
2023-01-11T21:03:04.6221023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6221147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6221427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6221562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6221877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6222183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6222458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6222580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6222852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6222987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6223165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6223436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6223560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6223843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6223977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6224152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6224437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6224735Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6224902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6225066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6225227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6225471Z STAGE:2023-01-11 20:49:33 5093:5093 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6225713Z STAGE:2023-01-11 20:49:33 5092:5092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6225951Z STAGE:2023-01-11 20:49:33 5094:5094 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6226194Z STAGE:2023-01-11 20:49:33 5093:5093 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6226477Z STAGE:2023-01-11 20:49:33 5092:5092 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6226781Z STAGE:2023-01-11 20:49:33 5093:5093 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6227036Z STAGE:2023-01-11 20:49:33 5092:5092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6227281Z STAGE:2023-01-11 20:49:33 5094:5094 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6227533Z STAGE:2023-01-11 20:49:33 5094:5094 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6228101Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6228255Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6228818Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6228969Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6229524Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6229667Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6229721Z ok (2.284s)
2023-01-11T21:03:04.6229727Z 
2023-01-11T21:03:04.6229927Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6230002Z Ran 1 test in 2.284s
2023-01-11T21:03:04.6230007Z 
2023-01-11T21:03:04.6230069Z OK
2023-01-11T21:03:04.6230074Z 
2023-01-11T21:03:04.6230159Z Generating XML reports...
2023-01-11T21:03:04.6230503Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204931.xml
2023-01-11T21:03:04.6230737Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6231012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6231137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6231408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6231544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6231551Z 
2023-01-11T21:03:04.6231622Z Running tests...
2023-01-11T21:03:04.6231818Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6232053Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5148
2023-01-11T21:03:04.6232208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5149
2023-01-11T21:03:04.6232361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5150
2023-01-11T21:03:04.6232635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6232745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6233025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6233159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6233378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6233676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6233798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6234075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6234209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6234384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6234640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6234761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6235044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6235178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6235353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6235654Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6235947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6236236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6236400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6236631Z STAGE:2023-01-11 20:49:37 5148:5148 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6236794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6236958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6237203Z STAGE:2023-01-11 20:49:37 5150:5150 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6237446Z STAGE:2023-01-11 20:49:37 5149:5149 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6237689Z STAGE:2023-01-11 20:49:37 5148:5148 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6238096Z STAGE:2023-01-11 20:49:37 5150:5150 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:37 5149:5149 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6238102Z 
2023-01-11T21:03:04.6238355Z STAGE:2023-01-11 20:49:37 5148:5148 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6238779Z STAGE:2023-01-11 20:49:37 5150:5150 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:37 5149:5149 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6238786Z 
2023-01-11T21:03:04.6239023Z STAGE:2023-01-11 20:49:37 5148:5148 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6239401Z STAGE:2023-01-11 20:49:37 5149:5149 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:49:37 5150:5150 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6239418Z 
2023-01-11T21:03:04.6239648Z STAGE:2023-01-11 20:49:37 5148:5148 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6240220Z STAGE:2023-01-11 20:49:37 5150:5150 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:37 5149:5149 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:37 5148:5148 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6240259Z 
2023-01-11T21:03:04.6240275Z 
2023-01-11T21:03:04.6240711Z STAGE:2023-01-11 20:49:37 5149:5149 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:37 5150:5150 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6240729Z 
2023-01-11T21:03:04.6240954Z STAGE:2023-01-11 20:49:37 5148:5148 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6241344Z STAGE:2023-01-11 20:49:37 5149:5149 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:49:37 5150:5150 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6241350Z 
2023-01-11T21:03:04.6241593Z STAGE:2023-01-11 20:49:37 5148:5148 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6241843Z STAGE:2023-01-11 20:49:37 5148:5148 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6242091Z STAGE:2023-01-11 20:49:37 5149:5149 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6242330Z STAGE:2023-01-11 20:49:37 5150:5150 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6242813Z STAGE:2023-01-11 20:49:37 5149:5149 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:37 5150:5150 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6242819Z 
2023-01-11T21:03:04.6242890Z ok (2.275s)
2023-01-11T21:03:04.6242895Z 
2023-01-11T21:03:04.6243094Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6243156Z Ran 1 test in 2.276s
2023-01-11T21:03:04.6243174Z 
2023-01-11T21:03:04.6243222Z OK
2023-01-11T21:03:04.6243226Z 
2023-01-11T21:03:04.6243314Z Generating XML reports...
2023-01-11T21:03:04.6243656Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204935.xml
2023-01-11T21:03:04.6243894Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6244172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6244296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6244578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6244711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6244717Z 
2023-01-11T21:03:04.6244776Z Running tests...
2023-01-11T21:03:04.6244974Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6245165Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s)
2023-01-11T21:03:04.6245173Z 
2023-01-11T21:03:04.6245369Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6245446Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6245451Z 
2023-01-11T21:03:04.6245527Z OK (skipped=1)
2023-01-11T21:03:04.6245533Z 
2023-01-11T21:03:04.6245617Z Generating XML reports...
2023-01-11T21:03:04.6245954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204939.xml
2023-01-11T21:03:04.6246183Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6246446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6246568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6246849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6247014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6247019Z 
2023-01-11T21:03:04.6247091Z Running tests...
2023-01-11T21:03:04.6247316Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6247516Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s)
2023-01-11T21:03:04.6247522Z 
2023-01-11T21:03:04.6247715Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6247789Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6247794Z 
2023-01-11T21:03:04.6247853Z OK (skipped=1)
2023-01-11T21:03:04.6247858Z 
2023-01-11T21:03:04.6247939Z Generating XML reports...
2023-01-11T21:03:04.6248276Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204941.xml
2023-01-11T21:03:04.6248506Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6248780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6248900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6249181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6249312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6249318Z 
2023-01-11T21:03:04.6249377Z Running tests...
2023-01-11T21:03:04.6249569Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6249804Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5224
2023-01-11T21:03:04.6249959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5225
2023-01-11T21:03:04.6250109Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5226
2023-01-11T21:03:04.6250382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6250506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6250786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6250920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6251084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6251352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6251474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6251752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6251890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6252066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6252337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6252457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6252724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6252856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6253029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6253328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6253654Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6253977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6254141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6254305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6254466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6254627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6254798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6254967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6255262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6255558Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6255847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6256090Z STAGE:2023-01-11 20:49:44 5224:5224 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6256330Z STAGE:2023-01-11 20:49:44 5225:5225 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6256568Z STAGE:2023-01-11 20:49:44 5226:5226 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6256812Z STAGE:2023-01-11 20:49:44 5226:5226 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6257039Z STAGE:2023-01-11 20:49:44 5224:5224 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6257287Z STAGE:2023-01-11 20:49:44 5225:5225 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6257542Z STAGE:2023-01-11 20:49:44 5226:5226 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6257792Z STAGE:2023-01-11 20:49:44 5224:5224 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6258045Z STAGE:2023-01-11 20:49:44 5225:5225 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6258438Z STAGE:2023-01-11 20:49:44 5226:5226 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:49:44 5224:5224 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6258443Z 
2023-01-11T21:03:04.6258680Z STAGE:2023-01-11 20:49:44 5225:5225 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6259083Z STAGE:2023-01-11 20:49:44 5226:5226 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:44 5224:5224 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6259092Z 
2023-01-11T21:03:04.6259335Z STAGE:2023-01-11 20:49:44 5225:5225 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6259758Z STAGE:2023-01-11 20:49:44 5224:5224 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:44 5226:5226 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6259763Z 
2023-01-11T21:03:04.6260015Z STAGE:2023-01-11 20:49:44 5225:5225 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6260241Z STAGE:2023-01-11 20:49:44 5224:5224 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6260474Z STAGE:2023-01-11 20:49:44 5226:5226 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6260744Z STAGE:2023-01-11 20:49:44 5225:5225 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6261012Z STAGE:2023-01-11 20:49:44 5225:5225 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6261412Z STAGE:2023-01-11 20:49:44 5226:5226 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:49:44 5224:5224 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6261418Z 
2023-01-11T21:03:04.6261784Z STAGE:2023-01-11 20:49:44 5225:5225 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6262236Z STAGE:2023-01-11 20:49:44 5224:5224 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:44 5226:5226 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6262242Z 
2023-01-11T21:03:04.6262310Z ok (2.280s)
2023-01-11T21:03:04.6262315Z 
2023-01-11T21:03:04.6262513Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6262592Z Ran 1 test in 2.281s
2023-01-11T21:03:04.6262597Z 
2023-01-11T21:03:04.6262645Z OK
2023-01-11T21:03:04.6262649Z 
2023-01-11T21:03:04.6262738Z Generating XML reports...
2023-01-11T21:03:04.6263084Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204943.xml
2023-01-11T21:03:04.6263316Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6263591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6263716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6264001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6264137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6264144Z 
2023-01-11T21:03:04.6264216Z Running tests...
2023-01-11T21:03:04.6264403Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6264635Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5289
2023-01-11T21:03:04.6264792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5290
2023-01-11T21:03:04.6264945Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5291
2023-01-11T21:03:04.6265223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6265344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6265620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6265755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6265923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6266194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6266315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6266594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6266729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6266906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6267285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6267406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6267689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6267870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6268115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6268419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6268719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6269011Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6269178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6269343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6269508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6269685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6269846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6270019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6270314Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6270610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6271014Z STAGE:2023-01-11 20:49:49 5290:5290 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:49:49 5291:5291 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6271022Z 
2023-01-11T21:03:04.6271272Z STAGE:2023-01-11 20:49:49 5290:5290 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6271517Z STAGE:2023-01-11 20:49:49 5291:5291 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6271947Z STAGE:2023-01-11 20:49:49 5290:5290 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:49:49 5291:5291 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6271952Z 
2023-01-11T21:03:04.6272194Z STAGE:2023-01-11 20:49:49 5291:5291 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6272435Z STAGE:2023-01-11 20:49:49 5290:5290 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6272662Z STAGE:2023-01-11 20:49:49 5291:5291 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6272899Z STAGE:2023-01-11 20:49:49 5290:5290 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6273157Z STAGE:2023-01-11 20:49:49 5291:5291 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6273413Z STAGE:2023-01-11 20:49:49 5290:5290 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6273714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6273782Z ok (2.378s)
2023-01-11T21:03:04.6273788Z 
2023-01-11T21:03:04.6273985Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6274061Z Ran 1 test in 2.378s
2023-01-11T21:03:04.6274066Z 
2023-01-11T21:03:04.6274128Z OK
2023-01-11T21:03:04.6274133Z 
2023-01-11T21:03:04.6274203Z Generating XML reports...
2023-01-11T21:03:04.6274547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204947.xml
2023-01-11T21:03:04.6274819Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6275135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6275262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6275546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6275682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6275687Z 
2023-01-11T21:03:04.6275759Z Running tests...
2023-01-11T21:03:04.6275955Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6276162Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2023-01-11T21:03:04.6276167Z 
2023-01-11T21:03:04.6276362Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6276440Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6276446Z 
2023-01-11T21:03:04.6276517Z OK (skipped=1)
2023-01-11T21:03:04.6276522Z 
2023-01-11T21:03:04.6276608Z Generating XML reports...
2023-01-11T21:03:04.6276954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204951.xml
2023-01-11T21:03:04.6277241Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6277519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6277630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6277913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6278047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6278054Z 
2023-01-11T21:03:04.6278126Z Running tests...
2023-01-11T21:03:04.6278326Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6278546Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2023-01-11T21:03:04.6278551Z 
2023-01-11T21:03:04.6278748Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6278823Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6278828Z 
2023-01-11T21:03:04.6278900Z OK (skipped=1)
2023-01-11T21:03:04.6278904Z 
2023-01-11T21:03:04.6278975Z Generating XML reports...
2023-01-11T21:03:04.6279313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204953.xml
2023-01-11T21:03:04.6279543Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6279818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6279944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6280229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6280363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6280368Z 
2023-01-11T21:03:04.6280439Z Running tests...
2023-01-11T21:03:04.6280634Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6280830Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.001s)
2023-01-11T21:03:04.6280850Z 
2023-01-11T21:03:04.6281030Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6281103Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6281107Z 
2023-01-11T21:03:04.6281178Z OK (skipped=1)
2023-01-11T21:03:04.6281217Z 
2023-01-11T21:03:04.6281299Z Generating XML reports...
2023-01-11T21:03:04.6281634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204954.xml
2023-01-11T21:03:04.6281894Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6282166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6282290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6282557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6282766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6282772Z 
2023-01-11T21:03:04.6282847Z Running tests...
2023-01-11T21:03:04.6283045Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6283265Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2023-01-11T21:03:04.6283271Z 
2023-01-11T21:03:04.6283469Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6283543Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6283548Z 
2023-01-11T21:03:04.6283619Z OK (skipped=1)
2023-01-11T21:03:04.6283624Z 
2023-01-11T21:03:04.6283706Z Generating XML reports...
2023-01-11T21:03:04.6284028Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204956.xml
2023-01-11T21:03:04.6284259Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6284530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6284656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6284939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6285077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6285085Z 
2023-01-11T21:03:04.6285161Z Running tests...
2023-01-11T21:03:04.6285353Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6285582Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5390
2023-01-11T21:03:04.6285743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5391
2023-01-11T21:03:04.6285894Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5392
2023-01-11T21:03:04.6286165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6286288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6286573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6286707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6286882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6287154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6287263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6287541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6287675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6287851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6288158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6288279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6288585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6288717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6288880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6289177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6289474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6289769Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6289933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6290102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6290262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6290330Z ok (2.176s)
2023-01-11T21:03:04.6290336Z 
2023-01-11T21:03:04.6290532Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6290594Z Ran 1 test in 2.176s
2023-01-11T21:03:04.6290611Z 
2023-01-11T21:03:04.6290659Z OK
2023-01-11T21:03:04.6290664Z 
2023-01-11T21:03:04.6290747Z Generating XML reports...
2023-01-11T21:03:04.6291088Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204958.xml
2023-01-11T21:03:04.6291319Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6291595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6291722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6292003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6292136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6292142Z 
2023-01-11T21:03:04.6292201Z Running tests...
2023-01-11T21:03:04.6292395Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6292637Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5443
2023-01-11T21:03:04.6292792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5444
2023-01-11T21:03:04.6292941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5445
2023-01-11T21:03:04.6293215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6293340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6293621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6293753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6293916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6294186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6294307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6294585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6294746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6294949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6295221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6295342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6295609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6295740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6295911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6296210Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6296510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6296803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6296968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6297128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6297289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6297449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6297623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6297914Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6298090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6298386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6298678Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6298851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:03:04.6299021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T21:03:04.6299193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:03:04.6299471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2023-01-11T21:03:04.6299763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2023-01-11T21:03:04.6300054Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2023-01-11T21:03:04.6300225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:03:04.6300393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:03:04.6300561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T21:03:04.6300852Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2023-01-11T21:03:04.6301143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2023-01-11T21:03:04.6301461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2023-01-11T21:03:04.6301527Z ok (2.380s)
2023-01-11T21:03:04.6301533Z 
2023-01-11T21:03:04.6301921Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6302005Z Ran 1 test in 2.381s
2023-01-11T21:03:04.6302010Z 
2023-01-11T21:03:04.6302072Z OK
2023-01-11T21:03:04.6302077Z 
2023-01-11T21:03:04.6302163Z Generating XML reports...
2023-01-11T21:03:04.6302511Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205002.xml
2023-01-11T21:03:04.6302748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6303025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6303148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6303420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6303557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6303562Z 
2023-01-11T21:03:04.6303636Z Running tests...
2023-01-11T21:03:04.6303835Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6304029Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2023-01-11T21:03:04.6304035Z 
2023-01-11T21:03:04.6304228Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6304303Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6304308Z 
2023-01-11T21:03:04.6304378Z OK (skipped=1)
2023-01-11T21:03:04.6304383Z 
2023-01-11T21:03:04.6304466Z Generating XML reports...
2023-01-11T21:03:04.6304795Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205006.xml
2023-01-11T21:03:04.6305030Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6327774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6328002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6328363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6328498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6328506Z 
2023-01-11T21:03:04.6328570Z Running tests...
2023-01-11T21:03:04.6328763Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6329008Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5542
2023-01-11T21:03:04.6329161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5543
2023-01-11T21:03:04.6329304Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5544
2023-01-11T21:03:04.6329575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6329690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6329963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6330088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6330259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6330524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6330638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6331059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6331223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6331393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6331655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6331764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6332034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6332163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6332337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6332642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6332942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6333236Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6333402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6333554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6333714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6333890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6334064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6334236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6334531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6334825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6335113Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6335360Z STAGE:2023-01-11 20:50:09 5542:5542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6335603Z STAGE:2023-01-11 20:50:09 5543:5543 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6335831Z STAGE:2023-01-11 20:50:09 5544:5544 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6336083Z STAGE:2023-01-11 20:50:09 5542:5542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6336493Z STAGE:2023-01-11 20:50:09 5544:5544 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:09 5543:5543 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6336500Z 
2023-01-11T21:03:04.6337095Z STAGE:2023-01-11 20:50:09 5542:5542 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:09 5544:5544 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:09 5543:5543 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6337101Z 
2023-01-11T21:03:04.6337105Z 
2023-01-11T21:03:04.6337342Z STAGE:2023-01-11 20:50:09 5544:5544 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6337728Z STAGE:2023-01-11 20:50:09 5543:5543 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:09 5542:5542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6337765Z 
2023-01-11T21:03:04.6338041Z STAGE:2023-01-11 20:50:09 5544:5544 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6338448Z STAGE:2023-01-11 20:50:09 5542:5542 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:09 5543:5543 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6338454Z 
2023-01-11T21:03:04.6338710Z STAGE:2023-01-11 20:50:09 5544:5544 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6338964Z STAGE:2023-01-11 20:50:09 5542:5542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6339211Z STAGE:2023-01-11 20:50:09 5543:5543 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6339448Z STAGE:2023-01-11 20:50:09 5544:5544 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6339675Z STAGE:2023-01-11 20:50:09 5543:5543 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6339911Z STAGE:2023-01-11 20:50:09 5542:5542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6340312Z STAGE:2023-01-11 20:50:09 5542:5542 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:09 5544:5544 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6340319Z 
2023-01-11T21:03:04.6340563Z STAGE:2023-01-11 20:50:09 5543:5543 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6340817Z STAGE:2023-01-11 20:50:09 5542:5542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6341067Z STAGE:2023-01-11 20:50:09 5544:5544 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6341316Z STAGE:2023-01-11 20:50:09 5543:5543 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6342133Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6342292Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6342858Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6343007Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6343558Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6343695Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6343763Z ok (2.274s)
2023-01-11T21:03:04.6343769Z 
2023-01-11T21:03:04.6343968Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6344046Z Ran 1 test in 2.275s
2023-01-11T21:03:04.6344051Z 
2023-01-11T21:03:04.6344111Z OK
2023-01-11T21:03:04.6344116Z 
2023-01-11T21:03:04.6344200Z Generating XML reports...
2023-01-11T21:03:04.6344546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205008.xml
2023-01-11T21:03:04.6344778Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6345057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6345238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6345565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6345703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6345709Z 
2023-01-11T21:03:04.6345783Z Running tests...
2023-01-11T21:03:04.6345982Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6346233Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5607
2023-01-11T21:03:04.6346390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5608
2023-01-11T21:03:04.6346544Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5609
2023-01-11T21:03:04.6346805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6346932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6347217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6347351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6347531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6347805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6347928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6348210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6348344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6348511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6348811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6349086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6349208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6349491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6349624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6349796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6350096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6350395Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6350550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6350713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6350875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6351050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6351221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6351394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6351688Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6352043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6352335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6352571Z STAGE:2023-01-11 20:50:14 5609:5609 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6352974Z STAGE:2023-01-11 20:50:14 5607:5607 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:14 5608:5608 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6352979Z 
2023-01-11T21:03:04.6353227Z STAGE:2023-01-11 20:50:14 5609:5609 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6353469Z STAGE:2023-01-11 20:50:14 5607:5607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6353714Z STAGE:2023-01-11 20:50:14 5608:5608 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6353973Z STAGE:2023-01-11 20:50:14 5609:5609 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6354397Z STAGE:2023-01-11 20:50:14 5608:5608 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:14 5607:5607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6354404Z 
2023-01-11T21:03:04.6354642Z STAGE:2023-01-11 20:50:14 5609:5609 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6355037Z STAGE:2023-01-11 20:50:14 5608:5608 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:14 5607:5607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6355042Z 
2023-01-11T21:03:04.6355283Z STAGE:2023-01-11 20:50:14 5608:5608 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6355539Z STAGE:2023-01-11 20:50:14 5608:5608 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6355932Z STAGE:2023-01-11 20:50:14 5609:5609 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:14 5607:5607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6355950Z 
2023-01-11T21:03:04.6356358Z STAGE:2023-01-11 20:50:14 5609:5609 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:14 5607:5607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6356377Z 
2023-01-11T21:03:04.6356600Z STAGE:2023-01-11 20:50:14 5608:5608 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6356991Z STAGE:2023-01-11 20:50:14 5609:5609 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:14 5607:5607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6356998Z 
2023-01-11T21:03:04.6357240Z STAGE:2023-01-11 20:50:14 5607:5607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6357646Z STAGE:2023-01-11 20:50:14 5608:5608 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:14 5609:5609 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6357652Z 
2023-01-11T21:03:04.6357905Z STAGE:2023-01-11 20:50:14 5607:5607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6358326Z STAGE:2023-01-11 20:50:14 5608:5608 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:14 5609:5609 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6358331Z 
2023-01-11T21:03:04.6358897Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6359079Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6359667Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6359818Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6360376Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6360522Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6360592Z ok (2.273s)
2023-01-11T21:03:04.6360597Z 
2023-01-11T21:03:04.6360786Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6360863Z Ran 1 test in 2.274s
2023-01-11T21:03:04.6360868Z 
2023-01-11T21:03:04.6360928Z OK
2023-01-11T21:03:04.6360933Z 
2023-01-11T21:03:04.6361019Z Generating XML reports...
2023-01-11T21:03:04.6361363Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205012.xml
2023-01-11T21:03:04.6361597Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6361874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6361998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6362265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6362402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6362407Z 
2023-01-11T21:03:04.6362479Z Running tests...
2023-01-11T21:03:04.6362770Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6363029Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5672
2023-01-11T21:03:04.6363186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5673
2023-01-11T21:03:04.6363338Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5674
2023-01-11T21:03:04.6363613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6363736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6364005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6364145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6364324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6364596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6364719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6365000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6365137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6365315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6365615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6365913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6366064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6366348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6366482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6366657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6366952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6367249Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6367415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6367578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6367730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6367904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6368075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6368248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6368543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6368834Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6369126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6369373Z STAGE:2023-01-11 20:50:18 5674:5674 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6369617Z STAGE:2023-01-11 20:50:18 5672:5672 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6369846Z STAGE:2023-01-11 20:50:18 5673:5673 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6370092Z STAGE:2023-01-11 20:50:18 5674:5674 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6370331Z STAGE:2023-01-11 20:50:18 5672:5672 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6370575Z STAGE:2023-01-11 20:50:18 5673:5673 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6370833Z STAGE:2023-01-11 20:50:18 5674:5674 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6371089Z STAGE:2023-01-11 20:50:18 5672:5672 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6371343Z STAGE:2023-01-11 20:50:18 5673:5673 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6371581Z STAGE:2023-01-11 20:50:18 5674:5674 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6371975Z STAGE:2023-01-11 20:50:18 5672:5672 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:18 5673:5673 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6371981Z 
2023-01-11T21:03:04.6372387Z STAGE:2023-01-11 20:50:18 5672:5672 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:18 5673:5673 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6372393Z 
2023-01-11T21:03:04.6372624Z STAGE:2023-01-11 20:50:18 5674:5674 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6373109Z STAGE:2023-01-11 20:50:18 5673:5673 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:18 5672:5672 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6373115Z 
2023-01-11T21:03:04.6373368Z STAGE:2023-01-11 20:50:18 5674:5674 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6373606Z STAGE:2023-01-11 20:50:18 5673:5673 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6373842Z STAGE:2023-01-11 20:50:18 5672:5672 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6374081Z STAGE:2023-01-11 20:50:18 5674:5674 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6374323Z STAGE:2023-01-11 20:50:18 5673:5673 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6374724Z STAGE:2023-01-11 20:50:18 5674:5674 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:18 5672:5672 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6374732Z 
2023-01-11T21:03:04.6374985Z STAGE:2023-01-11 20:50:18 5673:5673 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6375237Z STAGE:2023-01-11 20:50:18 5672:5672 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6375489Z STAGE:2023-01-11 20:50:18 5674:5674 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6376045Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6376196Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6376757Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6376908Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6377465Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6377612Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6377681Z ok (2.276s)
2023-01-11T21:03:04.6377685Z 
2023-01-11T21:03:04.6377884Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6377960Z Ran 1 test in 2.276s
2023-01-11T21:03:04.6377967Z 
2023-01-11T21:03:04.6378028Z OK
2023-01-11T21:03:04.6378033Z 
2023-01-11T21:03:04.6378104Z Generating XML reports...
2023-01-11T21:03:04.6378452Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205016.xml
2023-01-11T21:03:04.6378685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6378962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6379087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6379372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6379508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6379513Z 
2023-01-11T21:03:04.6379585Z Running tests...
2023-01-11T21:03:04.6379811Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6380078Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5737
2023-01-11T21:03:04.6380236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5738
2023-01-11T21:03:04.6380389Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5739
2023-01-11T21:03:04.6380661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6380785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6381067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6381203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6381382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6381762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6381901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6382188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6382323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6382502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6382775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6382900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6383183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6383320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6383484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6383783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6384080Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6384375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6384544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6384707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6384871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6385046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6385220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6385378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6385675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6385966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6386256Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6386501Z STAGE:2023-01-11 20:50:22 5738:5738 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6387000Z STAGE:2023-01-11 20:50:22 5739:5739 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:22 5737:5737 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6387007Z 
2023-01-11T21:03:04.6387258Z STAGE:2023-01-11 20:50:22 5738:5738 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6387501Z STAGE:2023-01-11 20:50:22 5737:5737 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6387747Z STAGE:2023-01-11 20:50:22 5739:5739 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6388001Z STAGE:2023-01-11 20:50:22 5738:5738 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6388240Z STAGE:2023-01-11 20:50:22 5737:5737 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6388495Z STAGE:2023-01-11 20:50:22 5739:5739 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6388738Z STAGE:2023-01-11 20:50:22 5738:5738 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6389136Z STAGE:2023-01-11 20:50:22 5739:5739 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:22 5737:5737 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6389142Z 
2023-01-11T21:03:04.6389386Z STAGE:2023-01-11 20:50:22 5739:5739 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6389627Z STAGE:2023-01-11 20:50:22 5737:5737 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6389864Z STAGE:2023-01-11 20:50:22 5738:5738 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6390118Z STAGE:2023-01-11 20:50:22 5739:5739 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6390369Z STAGE:2023-01-11 20:50:22 5737:5737 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6390624Z STAGE:2023-01-11 20:50:22 5738:5738 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6390852Z STAGE:2023-01-11 20:50:22 5739:5739 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6391243Z STAGE:2023-01-11 20:50:22 5738:5738 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:22 5737:5737 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6391248Z 
2023-01-11T21:03:04.6391656Z STAGE:2023-01-11 20:50:22 5737:5737 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:22 5738:5738 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6391661Z 
2023-01-11T21:03:04.6391903Z STAGE:2023-01-11 20:50:22 5739:5739 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6392328Z STAGE:2023-01-11 20:50:22 5737:5737 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:22 5738:5738 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6392335Z 
2023-01-11T21:03:04.6392591Z STAGE:2023-01-11 20:50:22 5739:5739 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6393155Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6393305Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6393863Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6394053Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6394682Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6394831Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6394900Z ok (2.278s)
2023-01-11T21:03:04.6394905Z 
2023-01-11T21:03:04.6395092Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6395167Z Ran 1 test in 2.278s
2023-01-11T21:03:04.6395173Z 
2023-01-11T21:03:04.6395234Z OK
2023-01-11T21:03:04.6395239Z 
2023-01-11T21:03:04.6395323Z Generating XML reports...
2023-01-11T21:03:04.6395667Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205020.xml
2023-01-11T21:03:04.6395905Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6396186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6396311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6396576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6396714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6396719Z 
2023-01-11T21:03:04.6396792Z Running tests...
2023-01-11T21:03:04.6396991Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6397240Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5802
2023-01-11T21:03:04.6397398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5803
2023-01-11T21:03:04.6397549Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5804
2023-01-11T21:03:04.6397824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6397947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6398215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6398352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6398528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6398800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6398922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6399203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6399338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6399512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6399782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6399894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6400176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6400312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6400487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6400823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6401150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6401449Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6401611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6401776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6401924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6402098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6402272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6402445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6402814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6403115Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6403518Z STAGE:2023-01-11 20:50:26 5803:5803 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:26 5804:5804 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6403524Z 
2023-01-11T21:03:04.6403773Z STAGE:2023-01-11 20:50:26 5804:5804 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6404017Z STAGE:2023-01-11 20:50:26 5803:5803 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6404262Z STAGE:2023-01-11 20:50:26 5804:5804 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6404515Z STAGE:2023-01-11 20:50:26 5803:5803 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6404755Z STAGE:2023-01-11 20:50:26 5804:5804 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6404996Z STAGE:2023-01-11 20:50:26 5803:5803 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6405408Z STAGE:2023-01-11 20:50:26 5804:5804 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:26 5803:5803 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6405413Z 
2023-01-11T21:03:04.6405833Z STAGE:2023-01-11 20:50:26 5803:5803 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:26 5804:5804 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6405841Z 
2023-01-11T21:03:04.6406139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6406710Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6406863Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6407417Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6407569Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6407672Z ok (2.282s)
2023-01-11T21:03:04.6407677Z 
2023-01-11T21:03:04.6407867Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6407943Z Ran 1 test in 2.282s
2023-01-11T21:03:04.6407976Z 
2023-01-11T21:03:04.6408038Z OK
2023-01-11T21:03:04.6408044Z 
2023-01-11T21:03:04.6408127Z Generating XML reports...
2023-01-11T21:03:04.6408469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205024.xml
2023-01-11T21:03:04.6408706Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6408980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6409104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6409386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6409510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6409515Z 
2023-01-11T21:03:04.6409588Z Running tests...
2023-01-11T21:03:04.6409789Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6410034Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5863
2023-01-11T21:03:04.6410193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5864
2023-01-11T21:03:04.6410344Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5865
2023-01-11T21:03:04.6410620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6410742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6411011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6411147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6411324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6411598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6411720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6412004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6412136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6412314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6412584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6412696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6412976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6413109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6413285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6413587Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6413886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6414182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6414350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6414550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6414725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6414900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6415072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6415244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6415543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6415836Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6416084Z STAGE:2023-01-11 20:50:30 5864:5864 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6416328Z STAGE:2023-01-11 20:50:30 5865:5865 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6416580Z STAGE:2023-01-11 20:50:30 5864:5864 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6416806Z STAGE:2023-01-11 20:50:30 5865:5865 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6417061Z STAGE:2023-01-11 20:50:30 5864:5864 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6417310Z STAGE:2023-01-11 20:50:30 5865:5865 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6417546Z STAGE:2023-01-11 20:50:30 5864:5864 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6417783Z STAGE:2023-01-11 20:50:30 5865:5865 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6418027Z STAGE:2023-01-11 20:50:30 5864:5864 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6418267Z STAGE:2023-01-11 20:50:30 5865:5865 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6418523Z STAGE:2023-01-11 20:50:30 5864:5864 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6418775Z STAGE:2023-01-11 20:50:30 5865:5865 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6419060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6419624Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6419773Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6420333Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6420482Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6420550Z ok (2.281s)
2023-01-11T21:03:04.6420556Z 
2023-01-11T21:03:04.6420754Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6420830Z Ran 1 test in 2.281s
2023-01-11T21:03:04.6420835Z 
2023-01-11T21:03:04.6420895Z OK
2023-01-11T21:03:04.6420900Z 
2023-01-11T21:03:04.6420982Z Generating XML reports...
2023-01-11T21:03:04.6421310Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205028.xml
2023-01-11T21:03:04.6421542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6421980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6422158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6422450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6422587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6422593Z 
2023-01-11T21:03:04.6422667Z Running tests...
2023-01-11T21:03:04.6422866Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6423103Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5924
2023-01-11T21:03:04.6423258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5925
2023-01-11T21:03:04.6423412Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5926
2023-01-11T21:03:04.6423690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6423813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6424099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6424240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6424419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6424690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6424801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6425080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6425216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6425392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6425664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6425786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6426065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6426197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6426368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6426655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6426954Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6427248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6427414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6427579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6427742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6427916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6428087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6428258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6428606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6428904Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6429149Z STAGE:2023-01-11 20:50:34 5925:5925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6429391Z STAGE:2023-01-11 20:50:34 5926:5926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6429638Z STAGE:2023-01-11 20:50:34 5926:5926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6429877Z STAGE:2023-01-11 20:50:34 5925:5925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6430130Z STAGE:2023-01-11 20:50:34 5926:5926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6430386Z STAGE:2023-01-11 20:50:34 5925:5925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6430624Z STAGE:2023-01-11 20:50:34 5925:5925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6430849Z STAGE:2023-01-11 20:50:34 5926:5926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6431092Z STAGE:2023-01-11 20:50:34 5925:5925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6431330Z STAGE:2023-01-11 20:50:34 5926:5926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6431583Z STAGE:2023-01-11 20:50:34 5925:5925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6431831Z STAGE:2023-01-11 20:50:34 5926:5926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6432133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6432700Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6432851Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6433406Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6433558Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6433612Z ok (2.374s)
2023-01-11T21:03:04.6433618Z 
2023-01-11T21:03:04.6433819Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6433896Z Ran 1 test in 2.375s
2023-01-11T21:03:04.6433902Z 
2023-01-11T21:03:04.6433961Z OK
2023-01-11T21:03:04.6433966Z 
2023-01-11T21:03:04.6434051Z Generating XML reports...
2023-01-11T21:03:04.6434395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205032.xml
2023-01-11T21:03:04.6434626Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6434901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6435012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6435293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6435428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6435466Z 
2023-01-11T21:03:04.6435539Z Running tests...
2023-01-11T21:03:04.6435737Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6436010Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5985
2023-01-11T21:03:04.6436168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5986
2023-01-11T21:03:04.6436322Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5987
2023-01-11T21:03:04.6436597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6436706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6436990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6437125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6437305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6437580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6437703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6437983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6438116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6438292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6438554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6438676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6438956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6439093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6439265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6439565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6439863Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6440155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6440319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6440471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6440637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6440814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6440986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6441155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6441448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6441742Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6441988Z STAGE:2023-01-11 20:50:38 5987:5987 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6442232Z STAGE:2023-01-11 20:50:38 5986:5986 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6442771Z STAGE:2023-01-11 20:50:38 5987:5987 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:38 5986:5986 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6442791Z 
2023-01-11T21:03:04.6443211Z STAGE:2023-01-11 20:50:38 5986:5986 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:38 5987:5987 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6443230Z 
2023-01-11T21:03:04.6443608Z STAGE:2023-01-11 20:50:38 5987:5987 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:38 5986:5986 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6443614Z 
2023-01-11T21:03:04.6443859Z STAGE:2023-01-11 20:50:38 5987:5987 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6444100Z STAGE:2023-01-11 20:50:38 5986:5986 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6444357Z STAGE:2023-01-11 20:50:38 5987:5987 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6444610Z STAGE:2023-01-11 20:50:38 5986:5986 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6444908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6445477Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6445634Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6446190Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6446344Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6446399Z ok (2.281s)
2023-01-11T21:03:04.6446418Z 
2023-01-11T21:03:04.6446605Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6446680Z Ran 1 test in 2.281s
2023-01-11T21:03:04.6446686Z 
2023-01-11T21:03:04.6446748Z OK
2023-01-11T21:03:04.6446753Z 
2023-01-11T21:03:04.6446838Z Generating XML reports...
2023-01-11T21:03:04.6447181Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205037.xml
2023-01-11T21:03:04.6447413Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6447689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6447816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6448088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6448225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6448230Z 
2023-01-11T21:03:04.6448302Z Running tests...
2023-01-11T21:03:04.6448499Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6448741Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6046
2023-01-11T21:03:04.6448895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6047
2023-01-11T21:03:04.6449047Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6048
2023-01-11T21:03:04.6449353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6449463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6449773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6449909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6450086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6450386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6450657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6450777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6451059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6451196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6451359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6451630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6451751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6452031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6452165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6452340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6452638Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6452935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6453099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6453250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6453412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6453655Z STAGE:2023-01-11 20:50:42 6046:6046 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6453898Z STAGE:2023-01-11 20:50:42 6048:6048 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6454136Z STAGE:2023-01-11 20:50:42 6047:6047 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6454547Z STAGE:2023-01-11 20:50:42 6046:6046 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:42 6047:6047 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6454555Z 
2023-01-11T21:03:04.6454801Z STAGE:2023-01-11 20:50:42 6048:6048 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6455399Z STAGE:2023-01-11 20:50:42 6047:6047 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:42 6046:6046 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:42 6048:6048 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6455406Z 
2023-01-11T21:03:04.6455410Z 
2023-01-11T21:03:04.6455647Z STAGE:2023-01-11 20:50:42 6048:6048 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6455884Z STAGE:2023-01-11 20:50:42 6047:6047 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6456153Z STAGE:2023-01-11 20:50:42 6046:6046 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6456579Z STAGE:2023-01-11 20:50:42 6047:6047 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:42 6048:6048 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6456826Z STAGE:2023-01-11 20:50:42 6046:6046 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6456832Z 
2023-01-11T21:03:04.6457426Z STAGE:2023-01-11 20:50:42 6047:6047 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:42 6048:6048 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:42 6046:6046 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6457431Z 
2023-01-11T21:03:04.6457450Z 
2023-01-11T21:03:04.6457980Z STAGE:2023-01-11 20:50:42 6047:6047 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:42 6048:6048 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:42 6046:6046 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6458001Z 
2023-01-11T21:03:04.6458008Z 
2023-01-11T21:03:04.6458238Z STAGE:2023-01-11 20:50:42 6047:6047 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6458478Z STAGE:2023-01-11 20:50:42 6046:6046 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6458714Z STAGE:2023-01-11 20:50:42 6048:6048 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6458967Z STAGE:2023-01-11 20:50:42 6047:6047 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6459221Z STAGE:2023-01-11 20:50:42 6046:6046 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6459472Z STAGE:2023-01-11 20:50:42 6048:6048 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6460041Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6460193Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6460751Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6460899Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6461457Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6461594Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6461771Z ok (2.278s)
2023-01-11T21:03:04.6461779Z 
2023-01-11T21:03:04.6462006Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6462083Z Ran 1 test in 2.278s
2023-01-11T21:03:04.6462088Z 
2023-01-11T21:03:04.6462150Z OK
2023-01-11T21:03:04.6462154Z 
2023-01-11T21:03:04.6462239Z Generating XML reports...
2023-01-11T21:03:04.6462582Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205041.xml
2023-01-11T21:03:04.6462814Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6463093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6463257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6463580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6463717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6463723Z 
2023-01-11T21:03:04.6463797Z Running tests...
2023-01-11T21:03:04.6463994Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6464259Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6102
2023-01-11T21:03:04.6464415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6103
2023-01-11T21:03:04.6464568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6104
2023-01-11T21:03:04.6464828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6464954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6465242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6465376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6465553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6465833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6465958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6466240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6466375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6466541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6466816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6466940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6467221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6467354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6467531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6467831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6468129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6468425Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6468578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6468741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6468901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6469466Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6469616Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6470211Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6470385Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6470932Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6471080Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6471147Z ok (2.181s)
2023-01-11T21:03:04.6471153Z 
2023-01-11T21:03:04.6471349Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6471412Z Ran 1 test in 2.181s
2023-01-11T21:03:04.6471420Z 
2023-01-11T21:03:04.6471480Z OK
2023-01-11T21:03:04.6471485Z 
2023-01-11T21:03:04.6471568Z Generating XML reports...
2023-01-11T21:03:04.6471912Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205045.xml
2023-01-11T21:03:04.6472147Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6472420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6472543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6472826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6472948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6472967Z 
2023-01-11T21:03:04.6473026Z Running tests...
2023-01-11T21:03:04.6473221Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6473464Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6155
2023-01-11T21:03:04.6473626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6156
2023-01-11T21:03:04.6473778Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6157
2023-01-11T21:03:04.6474051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6474174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6474456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6474577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6474753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6475026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6475151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6475432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6475569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6475743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6476041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6476296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6476418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6476730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6476891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6477064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6477362Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6477655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6477821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6477986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6478135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6478384Z STAGE:2023-01-11 20:50:51 6155:6155 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6478628Z STAGE:2023-01-11 20:50:51 6157:6157 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6478868Z STAGE:2023-01-11 20:50:51 6156:6156 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6479430Z STAGE:2023-01-11 20:50:51 6157:6157 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:51 6155:6155 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:51 6156:6156 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6479437Z 
2023-01-11T21:03:04.6479441Z 
2023-01-11T21:03:04.6480038Z STAGE:2023-01-11 20:50:51 6157:6157 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:51 6155:6155 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:51 6156:6156 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6480046Z 
2023-01-11T21:03:04.6480050Z 
2023-01-11T21:03:04.6480446Z STAGE:2023-01-11 20:50:51 6155:6155 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:51 6157:6157 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6480688Z STAGE:2023-01-11 20:50:51 6156:6156 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6480692Z 
2023-01-11T21:03:04.6480937Z STAGE:2023-01-11 20:50:51 6155:6155 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6481178Z STAGE:2023-01-11 20:50:51 6156:6156 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6481417Z STAGE:2023-01-11 20:50:51 6157:6157 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6481670Z STAGE:2023-01-11 20:50:51 6155:6155 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6481922Z STAGE:2023-01-11 20:50:51 6156:6156 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6482164Z STAGE:2023-01-11 20:50:51 6157:6157 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6482404Z STAGE:2023-01-11 20:50:51 6155:6155 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6482897Z STAGE:2023-01-11 20:50:51 6157:6157 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:51 6156:6156 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6482904Z 
2023-01-11T21:03:04.6483153Z STAGE:2023-01-11 20:50:51 6156:6156 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6483569Z STAGE:2023-01-11 20:50:51 6156:6156 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:51 6155:6155 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6483612Z 
2023-01-11T21:03:04.6483856Z STAGE:2023-01-11 20:50:51 6157:6157 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6484137Z STAGE:2023-01-11 20:50:51 6155:6155 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6484388Z STAGE:2023-01-11 20:50:51 6157:6157 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6484953Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6485106Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6485658Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6485809Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6486352Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6486497Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6486565Z ok (2.271s)
2023-01-11T21:03:04.6486571Z 
2023-01-11T21:03:04.6486771Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6486847Z Ran 1 test in 2.271s
2023-01-11T21:03:04.6486853Z 
2023-01-11T21:03:04.6486914Z OK
2023-01-11T21:03:04.6486919Z 
2023-01-11T21:03:04.6487005Z Generating XML reports...
2023-01-11T21:03:04.6487346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205049.xml
2023-01-11T21:03:04.6487583Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6487842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6487964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6488247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6488384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6488390Z 
2023-01-11T21:03:04.6488462Z Running tests...
2023-01-11T21:03:04.6488659Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6488903Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6211
2023-01-11T21:03:04.6489064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6212
2023-01-11T21:03:04.6489220Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6213
2023-01-11T21:03:04.6489481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6489604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6489884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6490018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6490195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6490470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6490638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6490946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6491068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6491244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6491520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6491645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6491926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6492059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6492237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6492537Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6492835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6493117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6493279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6493525Z STAGE:2023-01-11 20:50:55 6211:6211 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6493686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6493848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6494092Z STAGE:2023-01-11 20:50:55 6213:6213 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6494338Z STAGE:2023-01-11 20:50:55 6212:6212 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6494746Z STAGE:2023-01-11 20:50:55 6213:6213 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:55 6212:6212 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6494752Z 
2023-01-11T21:03:04.6494997Z STAGE:2023-01-11 20:50:55 6211:6211 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6495425Z STAGE:2023-01-11 20:50:55 6213:6213 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:55 6212:6212 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6495431Z 
2023-01-11T21:03:04.6495670Z STAGE:2023-01-11 20:50:55 6211:6211 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6496065Z STAGE:2023-01-11 20:50:55 6213:6213 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:55 6212:6212 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6496070Z 
2023-01-11T21:03:04.6496307Z STAGE:2023-01-11 20:50:55 6211:6211 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6496711Z STAGE:2023-01-11 20:50:55 6213:6213 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:55 6211:6211 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6496959Z STAGE:2023-01-11 20:50:55 6212:6212 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6496964Z 
2023-01-11T21:03:04.6497217Z STAGE:2023-01-11 20:50:55 6213:6213 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6497639Z STAGE:2023-01-11 20:50:55 6211:6211 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:55 6212:6212 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6497699Z 
2023-01-11T21:03:04.6497940Z STAGE:2023-01-11 20:50:55 6213:6213 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6498328Z STAGE:2023-01-11 20:50:55 6212:6212 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:55 6211:6211 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6498334Z 
2023-01-11T21:03:04.6498573Z STAGE:2023-01-11 20:50:55 6212:6212 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6498810Z STAGE:2023-01-11 20:50:55 6213:6213 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6499048Z STAGE:2023-01-11 20:50:55 6211:6211 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6499289Z STAGE:2023-01-11 20:50:55 6212:6212 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6499543Z STAGE:2023-01-11 20:50:55 6213:6213 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6499791Z STAGE:2023-01-11 20:50:55 6211:6211 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6500357Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6500510Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6501062Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6501210Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6501938Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6502088Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6502156Z ok (2.382s)
2023-01-11T21:03:04.6502162Z 
2023-01-11T21:03:04.6502362Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6502425Z Ran 1 test in 2.382s
2023-01-11T21:03:04.6502444Z 
2023-01-11T21:03:04.6502492Z OK
2023-01-11T21:03:04.6502496Z 
2023-01-11T21:03:04.6502584Z Generating XML reports...
2023-01-11T21:03:04.6502929Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205053.xml
2023-01-11T21:03:04.6503167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6503443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6503569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6503852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6503989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6503994Z 
2023-01-11T21:03:04.6504054Z Running tests...
2023-01-11T21:03:04.6504250Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6504490Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6267
2023-01-11T21:03:04.6504741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6268
2023-01-11T21:03:04.6504927Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6269
2023-01-11T21:03:04.6505204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6505330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6505613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6505735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6505914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6506186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6506315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6506598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6506734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6506912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6507184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6507308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6507573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6507707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6507884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6508185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6508482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6508776Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6508942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6509106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6509267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6509498Z STAGE:2023-01-11 20:50:59 6268:6268 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6509744Z STAGE:2023-01-11 20:50:59 6267:6267 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6509987Z STAGE:2023-01-11 20:50:59 6269:6269 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6510235Z STAGE:2023-01-11 20:50:59 6269:6269 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6510477Z STAGE:2023-01-11 20:50:59 6267:6267 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6510715Z STAGE:2023-01-11 20:50:59 6268:6268 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6510970Z STAGE:2023-01-11 20:50:59 6269:6269 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6511224Z STAGE:2023-01-11 20:50:59 6267:6267 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6511476Z STAGE:2023-01-11 20:50:59 6268:6268 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6511733Z STAGE:2023-01-11 20:50:59 6267:6267 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6511997Z STAGE:2023-01-11 20:50:59 6269:6269 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6512233Z STAGE:2023-01-11 20:50:59 6268:6268 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6512475Z STAGE:2023-01-11 20:50:59 6268:6268 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6512725Z STAGE:2023-01-11 20:50:59 6268:6268 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6513132Z STAGE:2023-01-11 20:50:59 6267:6267 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:59 6269:6269 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6513139Z 
2023-01-11T21:03:04.6513559Z STAGE:2023-01-11 20:50:59 6269:6269 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:59 6267:6267 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6513568Z 
2023-01-11T21:03:04.6513810Z STAGE:2023-01-11 20:50:59 6268:6268 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6514202Z STAGE:2023-01-11 20:50:59 6269:6269 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:50:59 6267:6267 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6514207Z 
2023-01-11T21:03:04.6514446Z STAGE:2023-01-11 20:50:59 6268:6268 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6514699Z STAGE:2023-01-11 20:50:59 6268:6268 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6515087Z STAGE:2023-01-11 20:50:59 6267:6267 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:50:59 6269:6269 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6515110Z 
2023-01-11T21:03:04.6515517Z STAGE:2023-01-11 20:50:59 6267:6267 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:50:59 6269:6269 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6515536Z 
2023-01-11T21:03:04.6516088Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6516239Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6516800Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6516950Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6517503Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:03:04.6517651Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:03:04.6517719Z ok (2.277s)
2023-01-11T21:03:04.6517725Z 
2023-01-11T21:03:04.6517924Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6517998Z Ran 1 test in 2.278s
2023-01-11T21:03:04.6518004Z 
2023-01-11T21:03:04.6518064Z OK
2023-01-11T21:03:04.6518069Z 
2023-01-11T21:03:04.6518139Z Generating XML reports...
2023-01-11T21:03:04.6518479Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205057.xml
2023-01-11T21:03:04.6518742Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6519045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6519171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6519453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6519589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6519594Z 
2023-01-11T21:03:04.6519667Z Running tests...
2023-01-11T21:03:04.6519865Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6520103Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6323
2023-01-11T21:03:04.6520261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6324
2023-01-11T21:03:04.6520413Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6325
2023-01-11T21:03:04.6520689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6520813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6521092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6521228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6521402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6521661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6521785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6522067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6522204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6522380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6522712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6522842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6523122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6523256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6523417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6523718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6524021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6524313Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6524479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6524645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6524808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6524876Z ok (2.183s)
2023-01-11T21:03:04.6524882Z 
2023-01-11T21:03:04.6525081Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6525142Z Ran 1 test in 2.184s
2023-01-11T21:03:04.6525183Z 
2023-01-11T21:03:04.6525246Z OK
2023-01-11T21:03:04.6525252Z 
2023-01-11T21:03:04.6525337Z Generating XML reports...
2023-01-11T21:03:04.6525708Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205101.xml
2023-01-11T21:03:04.6525942Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6526218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6526341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6526619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6526742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6526759Z 
2023-01-11T21:03:04.6526819Z Running tests...
2023-01-11T21:03:04.6527018Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6527258Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6376
2023-01-11T21:03:04.6527414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6377
2023-01-11T21:03:04.6527565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6378
2023-01-11T21:03:04.6527838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6527963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6528242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6528362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6528538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6528840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6529111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6529233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6529512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6529646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6529820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6530089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6530200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6530480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6530616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6530790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6531090Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6531386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6531550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6531713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6531871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6532070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6532270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6532444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6532742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6533036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6533327Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6533878Z STAGE:2023-01-11 20:51:07 6376:6376 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:07 6378:6378 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:07 6377:6377 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6533890Z 
2023-01-11T21:03:04.6533896Z 
2023-01-11T21:03:04.6534143Z STAGE:2023-01-11 20:51:07 6376:6376 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6534383Z STAGE:2023-01-11 20:51:07 6378:6378 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6534621Z STAGE:2023-01-11 20:51:07 6377:6377 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6534864Z STAGE:2023-01-11 20:51:07 6376:6376 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6535112Z STAGE:2023-01-11 20:51:07 6378:6378 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6535359Z STAGE:2023-01-11 20:51:07 6377:6377 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6535761Z STAGE:2023-01-11 20:51:07 6378:6378 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:07 6376:6376 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6535766Z 
2023-01-11T21:03:04.6536004Z STAGE:2023-01-11 20:51:07 6377:6377 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6536408Z STAGE:2023-01-11 20:51:07 6376:6376 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:07 6378:6378 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6536413Z 
2023-01-11T21:03:04.6536656Z STAGE:2023-01-11 20:51:07 6377:6377 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6537076Z STAGE:2023-01-11 20:51:07 6376:6376 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:51:07 6378:6378 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6537084Z 
2023-01-11T21:03:04.6537339Z STAGE:2023-01-11 20:51:07 6377:6377 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6537580Z STAGE:2023-01-11 20:51:07 6378:6378 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6537818Z STAGE:2023-01-11 20:51:07 6377:6377 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6538056Z STAGE:2023-01-11 20:51:07 6376:6376 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6538444Z STAGE:2023-01-11 20:51:07 6377:6377 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:07 6376:6376 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6538462Z 
2023-01-11T21:03:04.6538691Z STAGE:2023-01-11 20:51:07 6378:6378 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6539110Z STAGE:2023-01-11 20:51:07 6377:6377 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:51:07 6376:6376 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6539159Z 
2023-01-11T21:03:04.6539441Z STAGE:2023-01-11 20:51:07 6378:6378 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6539510Z ok (2.275s)
2023-01-11T21:03:04.6539515Z 
2023-01-11T21:03:04.6539713Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6539789Z Ran 1 test in 2.275s
2023-01-11T21:03:04.6539794Z 
2023-01-11T21:03:04.6539855Z OK
2023-01-11T21:03:04.6539860Z 
2023-01-11T21:03:04.6539943Z Generating XML reports...
2023-01-11T21:03:04.6540286Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205105.xml
2023-01-11T21:03:04.6540508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6540783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6540910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6541196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6541330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6541335Z 
2023-01-11T21:03:04.6541409Z Running tests...
2023-01-11T21:03:04.6541609Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6541954Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6441
2023-01-11T21:03:04.6542097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6442
2023-01-11T21:03:04.6542251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6443
2023-01-11T21:03:04.6542529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6542653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6542939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6543075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6543253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6543523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6543647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6543917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6544053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6544233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6544508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6544634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6544913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6545047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6545221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6545520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6545802Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6546183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6546349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6546508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6546671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6546846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6547022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6547193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6547483Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6547768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6548056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6548300Z STAGE:2023-01-11 20:51:11 6442:6442 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6548701Z STAGE:2023-01-11 20:51:11 6441:6441 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:11 6443:6443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6548707Z 
2023-01-11T21:03:04.6548954Z STAGE:2023-01-11 20:51:11 6442:6442 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6549195Z STAGE:2023-01-11 20:51:11 6441:6441 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6549437Z STAGE:2023-01-11 20:51:11 6443:6443 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6549698Z STAGE:2023-01-11 20:51:11 6442:6442 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6550129Z STAGE:2023-01-11 20:51:11 6441:6441 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:51:11 6443:6443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6550136Z 
2023-01-11T21:03:04.6550374Z STAGE:2023-01-11 20:51:11 6442:6442 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6550754Z STAGE:2023-01-11 20:51:11 6443:6443 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:11 6441:6441 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6550774Z 
2023-01-11T21:03:04.6551000Z STAGE:2023-01-11 20:51:11 6443:6443 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6551239Z STAGE:2023-01-11 20:51:11 6442:6442 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6551478Z STAGE:2023-01-11 20:51:11 6441:6441 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6551730Z STAGE:2023-01-11 20:51:11 6443:6443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6551982Z STAGE:2023-01-11 20:51:11 6442:6442 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6552229Z STAGE:2023-01-11 20:51:11 6441:6441 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6552470Z STAGE:2023-01-11 20:51:11 6443:6443 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6552707Z STAGE:2023-01-11 20:51:11 6442:6442 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6552943Z STAGE:2023-01-11 20:51:11 6441:6441 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6553393Z STAGE:2023-01-11 20:51:11 6443:6443 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:11 6441:6441 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6553413Z 
2023-01-11T21:03:04.6553646Z STAGE:2023-01-11 20:51:11 6442:6442 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6553898Z STAGE:2023-01-11 20:51:11 6443:6443 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6554149Z STAGE:2023-01-11 20:51:11 6441:6441 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6554401Z STAGE:2023-01-11 20:51:11 6442:6442 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6554468Z ok (2.283s)
2023-01-11T21:03:04.6554474Z 
2023-01-11T21:03:04.6554669Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6554747Z Ran 1 test in 2.283s
2023-01-11T21:03:04.6554753Z 
2023-01-11T21:03:04.6554814Z OK
2023-01-11T21:03:04.6554819Z 
2023-01-11T21:03:04.6554890Z Generating XML reports...
2023-01-11T21:03:04.6555235Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205109.xml
2023-01-11T21:03:04.6555468Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6555746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6555868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6556149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6556281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6556287Z 
2023-01-11T21:03:04.6556359Z Running tests...
2023-01-11T21:03:04.6556559Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6556795Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6506
2023-01-11T21:03:04.6556953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6507
2023-01-11T21:03:04.6557106Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6508
2023-01-11T21:03:04.6557379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6557500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6557782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6557918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6558095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6558355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6558477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6558754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6558890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6559067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6559365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6559635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6559789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6560098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6560220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6560394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6560688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6560984Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6561148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6561311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6561476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6561652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6561823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6561982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6562277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6562568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6562937Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6563190Z STAGE:2023-01-11 20:51:15 6506:6506 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6563438Z STAGE:2023-01-11 20:51:15 6507:6507 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6563680Z STAGE:2023-01-11 20:51:15 6508:6508 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6563927Z STAGE:2023-01-11 20:51:15 6506:6506 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6564168Z STAGE:2023-01-11 20:51:15 6508:6508 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6564399Z STAGE:2023-01-11 20:51:15 6507:6507 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6564650Z STAGE:2023-01-11 20:51:15 6506:6506 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6564906Z STAGE:2023-01-11 20:51:15 6508:6508 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6565159Z STAGE:2023-01-11 20:51:15 6507:6507 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6565398Z STAGE:2023-01-11 20:51:15 6506:6506 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6565639Z STAGE:2023-01-11 20:51:15 6507:6507 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6565880Z STAGE:2023-01-11 20:51:15 6508:6508 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6566124Z STAGE:2023-01-11 20:51:15 6507:6507 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6566363Z STAGE:2023-01-11 20:51:15 6506:6506 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6566589Z STAGE:2023-01-11 20:51:15 6508:6508 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6566840Z STAGE:2023-01-11 20:51:15 6507:6507 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6567092Z STAGE:2023-01-11 20:51:15 6506:6506 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6567377Z STAGE:2023-01-11 20:51:15 6508:6508 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6567642Z STAGE:2023-01-11 20:51:15 6507:6507 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6567880Z STAGE:2023-01-11 20:51:15 6506:6506 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6568117Z STAGE:2023-01-11 20:51:15 6508:6508 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6568360Z STAGE:2023-01-11 20:51:15 6507:6507 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6568597Z STAGE:2023-01-11 20:51:15 6508:6508 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6568820Z STAGE:2023-01-11 20:51:15 6506:6506 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6569071Z STAGE:2023-01-11 20:51:15 6507:6507 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6569322Z STAGE:2023-01-11 20:51:15 6508:6508 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6569575Z STAGE:2023-01-11 20:51:15 6506:6506 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6569644Z ok (2.277s)
2023-01-11T21:03:04.6569650Z 
2023-01-11T21:03:04.6569848Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6569923Z Ran 1 test in 2.277s
2023-01-11T21:03:04.6569928Z 
2023-01-11T21:03:04.6569988Z OK
2023-01-11T21:03:04.6569994Z 
2023-01-11T21:03:04.6570065Z Generating XML reports...
2023-01-11T21:03:04.6570408Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205114.xml
2023-01-11T21:03:04.6570641Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6570924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6571049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6571335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6571469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6571474Z 
2023-01-11T21:03:04.6571546Z Running tests...
2023-01-11T21:03:04.6571741Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6571965Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6571
2023-01-11T21:03:04.6572120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6572
2023-01-11T21:03:04.6572272Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6573
2023-01-11T21:03:04.6572548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6572673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6572955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6573089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6573266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6573565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6573825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6573945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6574255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6574418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6574596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6574867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6574994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6575272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6575392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6575570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6575869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6576172Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6576335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6576501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6576662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6576834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6577009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6577167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6577465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6577765Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6578059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6578306Z STAGE:2023-01-11 20:51:19 6572:6572 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6578548Z STAGE:2023-01-11 20:51:19 6573:6573 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6578786Z STAGE:2023-01-11 20:51:19 6571:6571 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6579031Z STAGE:2023-01-11 20:51:19 6571:6571 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6579271Z STAGE:2023-01-11 20:51:19 6573:6573 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6579501Z STAGE:2023-01-11 20:51:19 6572:6572 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6579927Z STAGE:2023-01-11 20:51:19 6571:6571 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:51:19 6573:6573 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6579934Z 
2023-01-11T21:03:04.6580187Z STAGE:2023-01-11 20:51:19 6572:6572 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6580730Z STAGE:2023-01-11 20:51:19 6573:6573 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:19 6572:6572 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:19 6571:6571 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6580736Z 
2023-01-11T21:03:04.6580741Z 
2023-01-11T21:03:04.6580988Z STAGE:2023-01-11 20:51:19 6571:6571 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6581454Z STAGE:2023-01-11 20:51:19 6573:6573 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:19 6572:6572 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6581461Z 
2023-01-11T21:03:04.6581833Z STAGE:2023-01-11 20:51:19 6571:6571 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6582088Z STAGE:2023-01-11 20:51:19 6573:6573 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6582346Z STAGE:2023-01-11 20:51:19 6572:6572 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6582584Z STAGE:2023-01-11 20:51:19 6573:6573 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6582976Z STAGE:2023-01-11 20:51:19 6572:6572 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:19 6571:6571 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6582985Z 
2023-01-11T21:03:04.6583393Z STAGE:2023-01-11 20:51:19 6572:6572 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:19 6573:6573 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6583637Z STAGE:2023-01-11 20:51:19 6571:6571 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6583643Z 
2023-01-11T21:03:04.6583893Z STAGE:2023-01-11 20:51:19 6572:6572 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6584130Z STAGE:2023-01-11 20:51:19 6573:6573 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6584378Z STAGE:2023-01-11 20:51:19 6571:6571 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6584446Z ok (2.173s)
2023-01-11T21:03:04.6584451Z 
2023-01-11T21:03:04.6584649Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6584728Z Ran 1 test in 2.173s
2023-01-11T21:03:04.6584734Z 
2023-01-11T21:03:04.6584794Z OK
2023-01-11T21:03:04.6584799Z 
2023-01-11T21:03:04.6584884Z Generating XML reports...
2023-01-11T21:03:04.6585227Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205118.xml
2023-01-11T21:03:04.6585447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6585721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6585846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6586128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6586266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6586271Z 
2023-01-11T21:03:04.6586347Z Running tests...
2023-01-11T21:03:04.6586547Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6586782Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6636
2023-01-11T21:03:04.6586936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6637
2023-01-11T21:03:04.6587077Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6638
2023-01-11T21:03:04.6587350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6587474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6587752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6587886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6588132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6588443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6588577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6588861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6588983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6589161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6589432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6589556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6589837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6589973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6590151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6590451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6590736Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6591034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6591199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6591361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6591525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6591702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6591874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6592044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6592337Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6592615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6593016Z STAGE:2023-01-11 20:51:23 6637:6637 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:23 6638:6638 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6593023Z 
2023-01-11T21:03:04.6593432Z STAGE:2023-01-11 20:51:23 6638:6638 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:23 6637:6637 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6593438Z 
2023-01-11T21:03:04.6593860Z STAGE:2023-01-11 20:51:23 6637:6637 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:51:23 6638:6638 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6593866Z 
2023-01-11T21:03:04.6594258Z STAGE:2023-01-11 20:51:23 6637:6637 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:23 6638:6638 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6594263Z 
2023-01-11T21:03:04.6594664Z STAGE:2023-01-11 20:51:23 6638:6638 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:23 6637:6637 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6594703Z 
2023-01-11T21:03:04.6595158Z STAGE:2023-01-11 20:51:23 6638:6638 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:51:23 6637:6637 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6595164Z 
2023-01-11T21:03:04.6595463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6595530Z ok (2.277s)
2023-01-11T21:03:04.6595535Z 
2023-01-11T21:03:04.6595734Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6595808Z Ran 1 test in 2.277s
2023-01-11T21:03:04.6595813Z 
2023-01-11T21:03:04.6595872Z OK
2023-01-11T21:03:04.6595877Z 
2023-01-11T21:03:04.6595960Z Generating XML reports...
2023-01-11T21:03:04.6596299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205122.xml
2023-01-11T21:03:04.6596523Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6596800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6596922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6597203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6597341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6597346Z 
2023-01-11T21:03:04.6597418Z Running tests...
2023-01-11T21:03:04.6597615Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6597850Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6697
2023-01-11T21:03:04.6598007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6698
2023-01-11T21:03:04.6598150Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6699
2023-01-11T21:03:04.6598427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6598552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6598834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6598968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6599144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6599416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6599538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6599805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6599942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6600120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6600390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6600512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6600791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6600925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6601100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6601398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6601744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6602045Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6602209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6602371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6602532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6602781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6602958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6603133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6603432Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6603714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6604117Z STAGE:2023-01-11 20:51:28 6698:6698 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:28 6699:6699 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6604123Z 
2023-01-11T21:03:04.6604528Z STAGE:2023-01-11 20:51:28 6699:6699 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:28 6698:6698 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6604534Z 
2023-01-11T21:03:04.6604958Z STAGE:2023-01-11 20:51:28 6698:6698 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:51:28 6699:6699 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6604967Z 
2023-01-11T21:03:04.6605210Z STAGE:2023-01-11 20:51:28 6699:6699 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6605452Z STAGE:2023-01-11 20:51:28 6698:6698 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6605698Z STAGE:2023-01-11 20:51:28 6698:6698 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6605937Z STAGE:2023-01-11 20:51:28 6699:6699 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6606194Z STAGE:2023-01-11 20:51:28 6698:6698 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6606444Z STAGE:2023-01-11 20:51:28 6699:6699 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6606742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6606799Z ok (2.276s)
2023-01-11T21:03:04.6606804Z 
2023-01-11T21:03:04.6607006Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6607082Z Ran 1 test in 2.276s
2023-01-11T21:03:04.6607087Z 
2023-01-11T21:03:04.6607152Z OK
2023-01-11T21:03:04.6607157Z 
2023-01-11T21:03:04.6607240Z Generating XML reports...
2023-01-11T21:03:04.6607584Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205126.xml
2023-01-11T21:03:04.6607814Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6608089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6608215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6608483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6608652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6608658Z 
2023-01-11T21:03:04.6608796Z Running tests...
2023-01-11T21:03:04.6608995Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6609238Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6758
2023-01-11T21:03:04.6609394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6759
2023-01-11T21:03:04.6609549Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6760
2023-01-11T21:03:04.6609823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6609932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6610216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6610351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6610529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6610801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6610926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6611207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6611340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6611517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6611777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6611904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6612188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6612322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6612502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6612802Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6613101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6613395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6613561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6613724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6613888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6614049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6614217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6614389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6614683Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6614978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6615255Z STAGE:2023-01-11 20:51:32 6760:6760 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6615526Z STAGE:2023-01-11 20:51:32 6759:6759 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6615762Z STAGE:2023-01-11 20:51:32 6759:6759 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6616003Z STAGE:2023-01-11 20:51:32 6760:6760 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6616256Z STAGE:2023-01-11 20:51:32 6759:6759 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6616504Z STAGE:2023-01-11 20:51:32 6760:6760 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6616741Z STAGE:2023-01-11 20:51:32 6759:6759 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6616975Z STAGE:2023-01-11 20:51:32 6760:6760 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6617221Z STAGE:2023-01-11 20:51:32 6760:6760 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6617463Z STAGE:2023-01-11 20:51:32 6759:6759 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6617716Z STAGE:2023-01-11 20:51:32 6760:6760 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6617954Z STAGE:2023-01-11 20:51:32 6759:6759 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6618255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6618322Z ok (2.377s)
2023-01-11T21:03:04.6618327Z 
2023-01-11T21:03:04.6618526Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6618601Z Ran 1 test in 2.377s
2023-01-11T21:03:04.6618607Z 
2023-01-11T21:03:04.6618668Z OK
2023-01-11T21:03:04.6618673Z 
2023-01-11T21:03:04.6618760Z Generating XML reports...
2023-01-11T21:03:04.6619102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205130.xml
2023-01-11T21:03:04.6619327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6619605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6619727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6620009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6620143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6620148Z 
2023-01-11T21:03:04.6620220Z Running tests...
2023-01-11T21:03:04.6620420Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6620656Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6819
2023-01-11T21:03:04.6620815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6820
2023-01-11T21:03:04.6620956Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6821
2023-01-11T21:03:04.6621228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6621349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6621630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6621861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6622041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6622316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6622487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6622789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6622927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6623101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6623377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6623500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6623779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6623913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6624091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6624390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6624673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6624965Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6625131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6625293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6625456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6625631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6625806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6625980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6626273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6626554Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6626798Z STAGE:2023-01-11 20:51:36 6820:6820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6627042Z STAGE:2023-01-11 20:51:36 6821:6821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6627288Z STAGE:2023-01-11 20:51:36 6821:6821 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6627530Z STAGE:2023-01-11 20:51:36 6820:6820 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6627787Z STAGE:2023-01-11 20:51:36 6821:6821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6628041Z STAGE:2023-01-11 20:51:36 6820:6820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6628437Z STAGE:2023-01-11 20:51:36 6820:6820 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:36 6821:6821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6628443Z 
2023-01-11T21:03:04.6628683Z STAGE:2023-01-11 20:51:36 6820:6820 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6628920Z STAGE:2023-01-11 20:51:36 6821:6821 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6629158Z STAGE:2023-01-11 20:51:36 6820:6820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6629439Z STAGE:2023-01-11 20:51:36 6821:6821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6629769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6629839Z ok (2.281s)
2023-01-11T21:03:04.6629844Z 
2023-01-11T21:03:04.6630045Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6630120Z Ran 1 test in 2.281s
2023-01-11T21:03:04.6630126Z 
2023-01-11T21:03:04.6630185Z OK
2023-01-11T21:03:04.6630190Z 
2023-01-11T21:03:04.6630274Z Generating XML reports...
2023-01-11T21:03:04.6630604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205134.xml
2023-01-11T21:03:04.6630839Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6631119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6631244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6631527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6631662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6631668Z 
2023-01-11T21:03:04.6631740Z Running tests...
2023-01-11T21:03:04.6631937Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6632166Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6880
2023-01-11T21:03:04.6632308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6881
2023-01-11T21:03:04.6632461Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6882
2023-01-11T21:03:04.6632735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6632861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6633146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6633282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6633458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6633730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6633851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6634115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6634249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6634428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6634729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6635001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6635124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6635404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6635536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6635696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6635992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6636336Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6636531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6636695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6636857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6637103Z STAGE:2023-01-11 20:51:40 6880:6880 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6637343Z STAGE:2023-01-11 20:51:40 6881:6881 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6637584Z STAGE:2023-01-11 20:51:40 6882:6882 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6637816Z STAGE:2023-01-11 20:51:40 6880:6880 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6638060Z STAGE:2023-01-11 20:51:40 6882:6882 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6638313Z STAGE:2023-01-11 20:51:40 6881:6881 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6638568Z STAGE:2023-01-11 20:51:40 6880:6880 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6638824Z STAGE:2023-01-11 20:51:40 6882:6882 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6639077Z STAGE:2023-01-11 20:51:40 6881:6881 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6639471Z STAGE:2023-01-11 20:51:40 6881:6881 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:40 6882:6882 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6639477Z 
2023-01-11T21:03:04.6639712Z STAGE:2023-01-11 20:51:40 6880:6880 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6640110Z STAGE:2023-01-11 20:51:40 6880:6880 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:40 6882:6882 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6640129Z 
2023-01-11T21:03:04.6640359Z STAGE:2023-01-11 20:51:40 6881:6881 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6640785Z STAGE:2023-01-11 20:51:40 6880:6880 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:51:40 6882:6882 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6640790Z 
2023-01-11T21:03:04.6641043Z STAGE:2023-01-11 20:51:40 6881:6881 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6641282Z STAGE:2023-01-11 20:51:40 6882:6882 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6641520Z STAGE:2023-01-11 20:51:40 6881:6881 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6641759Z STAGE:2023-01-11 20:51:40 6880:6880 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6642327Z STAGE:2023-01-11 20:51:40 6881:6881 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:40 6880:6880 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:40 6882:6882 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6642334Z 
2023-01-11T21:03:04.6642338Z 
2023-01-11T21:03:04.6642839Z STAGE:2023-01-11 20:51:40 6881:6881 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:51:40 6880:6880 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6643098Z STAGE:2023-01-11 20:51:40 6882:6882 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6643103Z 
2023-01-11T21:03:04.6643206Z ok (2.273s)
2023-01-11T21:03:04.6643211Z 
2023-01-11T21:03:04.6643413Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6643489Z Ran 1 test in 2.273s
2023-01-11T21:03:04.6643523Z 
2023-01-11T21:03:04.6643572Z OK
2023-01-11T21:03:04.6643577Z 
2023-01-11T21:03:04.6643664Z Generating XML reports...
2023-01-11T21:03:04.6644011Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205138.xml
2023-01-11T21:03:04.6644242Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6644519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6644643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6644926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6645063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6645068Z 
2023-01-11T21:03:04.6645142Z Running tests...
2023-01-11T21:03:04.6645328Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6645556Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6936
2023-01-11T21:03:04.6645714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6937
2023-01-11T21:03:04.6645868Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6938
2023-01-11T21:03:04.6646141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6646265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6646547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6646689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6646856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6647130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6647253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6647532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6647669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6647847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6648117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6648243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6648521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6648645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6648825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6649125Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6649424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6649717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6649884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6650074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6650262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6650512Z STAGE:2023-01-11 20:51:44 6938:6938 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6650742Z STAGE:2023-01-11 20:51:44 6936:6936 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6650980Z STAGE:2023-01-11 20:51:44 6937:6937 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6651228Z STAGE:2023-01-11 20:51:44 6936:6936 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6651640Z STAGE:2023-01-11 20:51:44 6937:6937 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:44 6938:6938 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6651646Z 
2023-01-11T21:03:04.6651906Z STAGE:2023-01-11 20:51:44 6936:6936 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6652328Z STAGE:2023-01-11 20:51:44 6937:6937 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:51:44 6938:6938 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6652334Z 
2023-01-11T21:03:04.6652570Z STAGE:2023-01-11 20:51:44 6937:6937 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6652961Z STAGE:2023-01-11 20:51:44 6936:6936 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:44 6938:6938 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6652967Z 
2023-01-11T21:03:04.6653370Z STAGE:2023-01-11 20:51:44 6936:6936 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:44 6937:6937 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6653375Z 
2023-01-11T21:03:04.6653624Z STAGE:2023-01-11 20:51:44 6938:6938 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6653878Z STAGE:2023-01-11 20:51:44 6936:6936 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6654131Z STAGE:2023-01-11 20:51:44 6937:6937 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6654370Z STAGE:2023-01-11 20:51:44 6938:6938 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6654759Z STAGE:2023-01-11 20:51:44 6937:6937 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:44 6936:6936 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6654764Z 
2023-01-11T21:03:04.6655001Z STAGE:2023-01-11 20:51:44 6938:6938 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6655242Z STAGE:2023-01-11 20:51:44 6937:6937 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6655648Z STAGE:2023-01-11 20:51:44 6936:6936 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:44 6938:6938 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6655654Z 
2023-01-11T21:03:04.6655904Z STAGE:2023-01-11 20:51:44 6937:6937 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6656150Z STAGE:2023-01-11 20:51:44 6936:6936 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6656399Z STAGE:2023-01-11 20:51:44 6938:6938 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6656466Z ok (2.275s)
2023-01-11T21:03:04.6656471Z 
2023-01-11T21:03:04.6656671Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6656736Z Ran 1 test in 2.276s
2023-01-11T21:03:04.6656742Z 
2023-01-11T21:03:04.6656802Z OK
2023-01-11T21:03:04.6656807Z 
2023-01-11T21:03:04.6656890Z Generating XML reports...
2023-01-11T21:03:04.6657263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205142.xml
2023-01-11T21:03:04.6657522Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6657804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6657927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6658211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6658334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6658350Z 
2023-01-11T21:03:04.6658410Z Running tests...
2023-01-11T21:03:04.6658605Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6658837Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6992
2023-01-11T21:03:04.6658993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6993
2023-01-11T21:03:04.6659148Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6994
2023-01-11T21:03:04.6659426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6659546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6659824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6659945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6660122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6660394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6660519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6660801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6660935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6661109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6661381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6661501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6661879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6662013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6662190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6662492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6662790Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6663088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6663254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6663417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6663576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6663660Z skip: CUDA is not available. (2.286s)
2023-01-11T21:03:04.6663721Z 
2023-01-11T21:03:04.6663918Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6663994Z Ran 1 test in 2.286s
2023-01-11T21:03:04.6664000Z 
2023-01-11T21:03:04.6664110Z OK (skipped=1)
2023-01-11T21:03:04.6664117Z 
2023-01-11T21:03:04.6664202Z Generating XML reports...
2023-01-11T21:03:04.6664545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205146.xml
2023-01-11T21:03:04.6664779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6665052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6665162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6665443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6665577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6665585Z 
2023-01-11T21:03:04.6665659Z Running tests...
2023-01-11T21:03:04.6665858Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6666098Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7045
2023-01-11T21:03:04.6666255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7046
2023-01-11T21:03:04.6666410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7047
2023-01-11T21:03:04.6666679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6666787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6667067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6667204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6667383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6667682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6667955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6668081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6668359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6668492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6668653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6668928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6669050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6669331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6669465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6669640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6669936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6670233Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6670384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6670585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6670773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6670871Z skip: CUDA is not available. (2.177s)
2023-01-11T21:03:04.6670876Z 
2023-01-11T21:03:04.6671076Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6671149Z Ran 1 test in 2.177s
2023-01-11T21:03:04.6671155Z 
2023-01-11T21:03:04.6671227Z OK (skipped=1)
2023-01-11T21:03:04.6671232Z 
2023-01-11T21:03:04.6671315Z Generating XML reports...
2023-01-11T21:03:04.6671657Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205150.xml
2023-01-11T21:03:04.6671878Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6672153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6672290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6672573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6672706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6672711Z 
2023-01-11T21:03:04.6672784Z Running tests...
2023-01-11T21:03:04.6672979Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6673213Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7098
2023-01-11T21:03:04.6673369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7099
2023-01-11T21:03:04.6673509Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7100
2023-01-11T21:03:04.6673782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6673907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6674188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6674322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6674498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6674770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6674891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6675158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6675292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6675469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6675742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6675868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6676145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6676277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6676449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6676748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6677031Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6677359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6677558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6677722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6677883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6678129Z STAGE:2023-01-11 20:51:56 7099:7099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6678371Z STAGE:2023-01-11 20:51:56 7100:7100 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6678609Z STAGE:2023-01-11 20:51:56 7098:7098 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6679017Z STAGE:2023-01-11 20:51:56 7098:7098 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:56 7099:7099 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6679026Z 
2023-01-11T21:03:04.6679261Z STAGE:2023-01-11 20:51:56 7100:7100 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6679511Z STAGE:2023-01-11 20:51:56 7099:7099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6679935Z STAGE:2023-01-11 20:51:56 7098:7098 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:51:56 7100:7100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6679941Z 
2023-01-11T21:03:04.6680178Z STAGE:2023-01-11 20:51:56 7099:7099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6680571Z STAGE:2023-01-11 20:51:56 7098:7098 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:56 7100:7100 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6680579Z 
2023-01-11T21:03:04.6680821Z STAGE:2023-01-11 20:51:56 7099:7099 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6681234Z STAGE:2023-01-11 20:51:56 7098:7098 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:56 7099:7099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6681241Z 
2023-01-11T21:03:04.6681487Z STAGE:2023-01-11 20:51:56 7100:7100 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6681907Z STAGE:2023-01-11 20:51:56 7100:7100 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:51:56 7098:7098 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6681913Z 
2023-01-11T21:03:04.6682149Z STAGE:2023-01-11 20:51:56 7099:7099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6682538Z STAGE:2023-01-11 20:51:56 7100:7100 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:51:56 7098:7098 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6682546Z 
2023-01-11T21:03:04.6682869Z STAGE:2023-01-11 20:51:56 7099:7099 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6683126Z STAGE:2023-01-11 20:51:56 7099:7099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6683530Z STAGE:2023-01-11 20:51:56 7100:7100 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:51:56 7098:7098 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6683536Z 
2023-01-11T21:03:04.6683941Z STAGE:2023-01-11 20:51:56 7098:7098 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:51:56 7100:7100 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6683959Z 
2023-01-11T21:03:04.6684014Z ok (2.280s)
2023-01-11T21:03:04.6684071Z 
2023-01-11T21:03:04.6684274Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6684351Z Ran 1 test in 2.280s
2023-01-11T21:03:04.6684356Z 
2023-01-11T21:03:04.6684419Z OK
2023-01-11T21:03:04.6684452Z 
2023-01-11T21:03:04.6684537Z Generating XML reports...
2023-01-11T21:03:04.6684886Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205154.xml
2023-01-11T21:03:04.6685116Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6685392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6685503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6685781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6685915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6685923Z 
2023-01-11T21:03:04.6685997Z Running tests...
2023-01-11T21:03:04.6686195Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6686432Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7154
2023-01-11T21:03:04.6686588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7155
2023-01-11T21:03:04.6686742Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7156
2023-01-11T21:03:04.6687017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6687126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6687412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6687548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6687724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6687998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6688118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6688397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6688531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6688707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6688966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6689085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6689364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6689500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6689673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6689970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6690265Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6690555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6690719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6690900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6691061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6691182Z skip: CUDA is not available. (2.278s)
2023-01-11T21:03:04.6691187Z 
2023-01-11T21:03:04.6691388Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6691462Z Ran 1 test in 2.279s
2023-01-11T21:03:04.6691467Z 
2023-01-11T21:03:04.6691536Z OK (skipped=1)
2023-01-11T21:03:04.6691541Z 
2023-01-11T21:03:04.6691624Z Generating XML reports...
2023-01-11T21:03:04.6691962Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205159.xml
2023-01-11T21:03:04.6692181Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6692453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6692576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6692861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6692994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6692999Z 
2023-01-11T21:03:04.6693071Z Running tests...
2023-01-11T21:03:04.6693266Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6693492Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7207
2023-01-11T21:03:04.6693646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7208
2023-01-11T21:03:04.6693784Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7209
2023-01-11T21:03:04.6694055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6694181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6694462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6694596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6694769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6695036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6695157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6695424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6695556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6695731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6696002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6696123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6696402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6696537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6696710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6697008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6697294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6697641Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6697835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6697994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6698157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6698403Z STAGE:2023-01-11 20:52:05 7207:7207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6698646Z STAGE:2023-01-11 20:52:05 7208:7208 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6698887Z STAGE:2023-01-11 20:52:05 7209:7209 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6699132Z STAGE:2023-01-11 20:52:05 7207:7207 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6699361Z STAGE:2023-01-11 20:52:05 7209:7209 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6699601Z STAGE:2023-01-11 20:52:05 7208:7208 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6700030Z STAGE:2023-01-11 20:52:05 7209:7209 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:52:05 7207:7207 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6700035Z 
2023-01-11T21:03:04.6700292Z STAGE:2023-01-11 20:52:05 7208:7208 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6700530Z STAGE:2023-01-11 20:52:05 7209:7209 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6700925Z STAGE:2023-01-11 20:52:05 7208:7208 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:52:05 7207:7207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6700933Z 
2023-01-11T21:03:04.6701501Z STAGE:2023-01-11 20:52:05 7207:7207 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:52:05 7208:7208 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:52:05 7209:7209 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6701508Z 
2023-01-11T21:03:04.6701512Z 
2023-01-11T21:03:04.6702215Z STAGE:2023-01-11 20:52:05 7207:7207 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:52:05 7209:7209 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:52:05 7208:7208 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6702221Z 
2023-01-11T21:03:04.6702226Z 
2023-01-11T21:03:04.6702464Z STAGE:2023-01-11 20:52:05 7208:7208 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6702704Z STAGE:2023-01-11 20:52:05 7209:7209 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6702943Z STAGE:2023-01-11 20:52:05 7207:7207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6703190Z STAGE:2023-01-11 20:52:05 7208:7208 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6703432Z STAGE:2023-01-11 20:52:05 7207:7207 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6703847Z STAGE:2023-01-11 20:52:05 7209:7209 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:52:05 7208:7208 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6703852Z 
2023-01-11T21:03:04.6704092Z STAGE:2023-01-11 20:52:05 7207:7207 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6704346Z STAGE:2023-01-11 20:52:05 7209:7209 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6704416Z ok (2.281s)
2023-01-11T21:03:04.6704421Z 
2023-01-11T21:03:04.6704673Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6704749Z Ran 1 test in 2.281s
2023-01-11T21:03:04.6704754Z 
2023-01-11T21:03:04.6704820Z OK
2023-01-11T21:03:04.6704859Z 
2023-01-11T21:03:04.6704947Z Generating XML reports...
2023-01-11T21:03:04.6705293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205203.xml
2023-01-11T21:03:04.6705527Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6705792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6705918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6706202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6706340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6706348Z 
2023-01-11T21:03:04.6706423Z Running tests...
2023-01-11T21:03:04.6706620Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6706855Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7263
2023-01-11T21:03:04.6707012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7264
2023-01-11T21:03:04.6707151Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7265
2023-01-11T21:03:04.6707429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6707554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6707835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6707970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6708151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6708424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6708549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6708834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6708955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6709136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6709435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6709708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6709833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6710116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6710252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6710428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6710727Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6711010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6711176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6711424Z STAGE:2023-01-11 20:52:09 7263:7263 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6711621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6711805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6712048Z STAGE:2023-01-11 20:52:09 7264:7264 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6712291Z STAGE:2023-01-11 20:52:09 7265:7265 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6712537Z STAGE:2023-01-11 20:52:09 7264:7264 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6712948Z STAGE:2023-01-11 20:52:09 7265:7265 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:52:09 7263:7263 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6712955Z 
2023-01-11T21:03:04.6713196Z STAGE:2023-01-11 20:52:09 7264:7264 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6713626Z STAGE:2023-01-11 20:52:09 7263:7263 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:52:09 7265:7265 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6713645Z 
2023-01-11T21:03:04.6713871Z STAGE:2023-01-11 20:52:09 7264:7264 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6714266Z STAGE:2023-01-11 20:52:09 7263:7263 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:52:09 7265:7265 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6714272Z 
2023-01-11T21:03:04.6714516Z STAGE:2023-01-11 20:52:09 7264:7264 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6714766Z STAGE:2023-01-11 20:52:09 7264:7264 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6715168Z STAGE:2023-01-11 20:52:09 7263:7263 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:52:09 7265:7265 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6715176Z 
2023-01-11T21:03:04.6715602Z STAGE:2023-01-11 20:52:09 7265:7265 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:52:09 7263:7263 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6715608Z 
2023-01-11T21:03:04.6715848Z STAGE:2023-01-11 20:52:09 7264:7264 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6716086Z STAGE:2023-01-11 20:52:09 7263:7263 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6716324Z STAGE:2023-01-11 20:52:09 7265:7265 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6716566Z STAGE:2023-01-11 20:52:09 7264:7264 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6716807Z STAGE:2023-01-11 20:52:09 7264:7264 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6717216Z STAGE:2023-01-11 20:52:09 7263:7263 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:52:09 7265:7265 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6717221Z 
2023-01-11T21:03:04.6717638Z STAGE:2023-01-11 20:52:09 7265:7265 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:52:09 7263:7263 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6717643Z 
2023-01-11T21:03:04.6717709Z ok (2.380s)
2023-01-11T21:03:04.6717715Z 
2023-01-11T21:03:04.6717911Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6717985Z Ran 1 test in 2.381s
2023-01-11T21:03:04.6717990Z 
2023-01-11T21:03:04.6718053Z OK
2023-01-11T21:03:04.6718057Z 
2023-01-11T21:03:04.6718143Z Generating XML reports...
2023-01-11T21:03:04.6718485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205207.xml
2023-01-11T21:03:04.6718823Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6719100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6719210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6719493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6719627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6719633Z 
2023-01-11T21:03:04.6719705Z Running tests...
2023-01-11T21:03:04.6719904Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6720142Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7319
2023-01-11T21:03:04.6720302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7320
2023-01-11T21:03:04.6720458Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7321
2023-01-11T21:03:04.6720719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6720841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6721124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6721257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6721435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6721710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6721836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6722120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6722254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6722418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6722783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6723060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6723184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6723466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6723600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6723775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6724078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6724376Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6724528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6724692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6724854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6725102Z STAGE:2023-01-11 20:52:13 7319:7319 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6725382Z STAGE:2023-01-11 20:52:13 7320:7320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6725650Z STAGE:2023-01-11 20:52:13 7321:7321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6725897Z STAGE:2023-01-11 20:52:13 7320:7320 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6726301Z STAGE:2023-01-11 20:52:13 7321:7321 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:52:13 7319:7319 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6726307Z 
2023-01-11T21:03:04.6726561Z STAGE:2023-01-11 20:52:13 7320:7320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6726971Z STAGE:2023-01-11 20:52:13 7321:7321 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:52:13 7319:7319 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6726992Z 
2023-01-11T21:03:04.6727221Z STAGE:2023-01-11 20:52:13 7320:7320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6727615Z STAGE:2023-01-11 20:52:13 7321:7321 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:52:13 7319:7319 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6727620Z 
2023-01-11T21:03:04.6728185Z STAGE:2023-01-11 20:52:13 7319:7319 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:52:13 7320:7320 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:52:13 7321:7321 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6728191Z 
2023-01-11T21:03:04.6728207Z 
2023-01-11T21:03:04.6728784Z STAGE:2023-01-11 20:52:13 7320:7320 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:52:13 7319:7319 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:52:13 7321:7321 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6728805Z 
2023-01-11T21:03:04.6728809Z 
2023-01-11T21:03:04.6729038Z STAGE:2023-01-11 20:52:13 7321:7321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6729426Z STAGE:2023-01-11 20:52:13 7319:7319 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:52:13 7320:7320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6729431Z 
2023-01-11T21:03:04.6729992Z STAGE:2023-01-11 20:52:13 7320:7320 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:52:13 7321:7321 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:52:13 7319:7319 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6729997Z 
2023-01-11T21:03:04.6730013Z 
2023-01-11T21:03:04.6730254Z STAGE:2023-01-11 20:52:13 7320:7320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6730674Z STAGE:2023-01-11 20:52:13 7321:7321 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:52:13 7319:7319 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6730679Z 
2023-01-11T21:03:04.6730744Z ok (2.281s)
2023-01-11T21:03:04.6730749Z 
2023-01-11T21:03:04.6730947Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6731021Z Ran 1 test in 2.281s
2023-01-11T21:03:04.6731026Z 
2023-01-11T21:03:04.6731087Z OK
2023-01-11T21:03:04.6731091Z 
2023-01-11T21:03:04.6731175Z Generating XML reports...
2023-01-11T21:03:04.6731515Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205211.xml
2023-01-11T21:03:04.6731746Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6732009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6732176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6732490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6732626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6732632Z 
2023-01-11T21:03:04.6732703Z Running tests...
2023-01-11T21:03:04.6732899Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6733131Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7375
2023-01-11T21:03:04.6733285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7376
2023-01-11T21:03:04.6733424Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7377
2023-01-11T21:03:04.6733695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6733820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6734102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6734237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6734415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6734687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6734813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6735092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6735212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6735391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6735663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6735784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6736060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6736191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6736364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6736661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6736958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6737240Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6737405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6737561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6737720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6737815Z skip: CUDA is not available. (2.178s)
2023-01-11T21:03:04.6737820Z 
2023-01-11T21:03:04.6738015Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6738089Z Ran 1 test in 2.178s
2023-01-11T21:03:04.6738094Z 
2023-01-11T21:03:04.6738164Z OK (skipped=1)
2023-01-11T21:03:04.6738168Z 
2023-01-11T21:03:04.6738239Z Generating XML reports...
2023-01-11T21:03:04.6738578Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205215.xml
2023-01-11T21:03:04.6738840Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6739143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6739267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6739548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6739683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6739688Z 
2023-01-11T21:03:04.6739760Z Running tests...
2023-01-11T21:03:04.6739956Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6740180Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7428
2023-01-11T21:03:04.6740338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7429
2023-01-11T21:03:04.6740490Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7430
2023-01-11T21:03:04.6740760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6740883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6741162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6741294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6741468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6741853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6741967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6742251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6742385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6742560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6742833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6742958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6743239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6743370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6743532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6743837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6744134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6744428Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6744593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6744756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6744916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6745011Z skip: CUDA is not available. (2.177s)
2023-01-11T21:03:04.6745016Z 
2023-01-11T21:03:04.6745211Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6745326Z Ran 1 test in 2.177s
2023-01-11T21:03:04.6745344Z 
2023-01-11T21:03:04.6745402Z OK (skipped=1)
2023-01-11T21:03:04.6745407Z 
2023-01-11T21:03:04.6745523Z Generating XML reports...
2023-01-11T21:03:04.6745868Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205219.xml
2023-01-11T21:03:04.6746100Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6746374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6746498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6746779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6746913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6746921Z 
2023-01-11T21:03:04.6746980Z Running tests...
2023-01-11T21:03:04.6747176Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6747418Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7481
2023-01-11T21:03:04.6747574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7482
2023-01-11T21:03:04.6747727Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7483
2023-01-11T21:03:04.6748000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6748122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6748400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6748522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6748699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6748972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6749094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6749379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6749513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6749690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6749961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6750084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6750352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6750487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6750661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6750960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6751255Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6751545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6751707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6751870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6752065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6752174Z skip: CUDA is not available. (2.177s)
2023-01-11T21:03:04.6752179Z 
2023-01-11T21:03:04.6752380Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6752455Z Ran 1 test in 2.177s
2023-01-11T21:03:04.6752460Z 
2023-01-11T21:03:04.6752530Z OK (skipped=1)
2023-01-11T21:03:04.6752535Z 
2023-01-11T21:03:04.6752617Z Generating XML reports...
2023-01-11T21:03:04.6752958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205223.xml
2023-01-11T21:03:04.6753190Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6753462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6753588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6753858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6753994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6754000Z 
2023-01-11T21:03:04.6754071Z Running tests...
2023-01-11T21:03:04.6754268Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6754443Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2023-01-11T21:03:04.6754449Z 
2023-01-11T21:03:04.6754644Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6754717Z Ran 1 test in 0.001s
2023-01-11T21:03:04.6754723Z 
2023-01-11T21:03:04.6754792Z OK (skipped=1)
2023-01-11T21:03:04.6754797Z 
2023-01-11T21:03:04.6754868Z Generating XML reports...
2023-01-11T21:03:04.6755201Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205227.xml
2023-01-11T21:03:04.6755432Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6755710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6755833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6756110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6756244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6756249Z 
2023-01-11T21:03:04.6756322Z Running tests...
2023-01-11T21:03:04.6756518Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6756689Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2023-01-11T21:03:04.6756696Z 
2023-01-11T21:03:04.6756886Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6756959Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6756965Z 
2023-01-11T21:03:04.6757037Z OK (skipped=1)
2023-01-11T21:03:04.6757042Z 
2023-01-11T21:03:04.6757127Z Generating XML reports...
2023-01-11T21:03:04.6757464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205229.xml
2023-01-11T21:03:04.6757692Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6757963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6758085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6758352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6758520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6758525Z 
2023-01-11T21:03:04.6758596Z Running tests...
2023-01-11T21:03:04.6758825Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6759012Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2023-01-11T21:03:04.6759017Z 
2023-01-11T21:03:04.6759207Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6759282Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6759287Z 
2023-01-11T21:03:04.6759359Z OK (skipped=1)
2023-01-11T21:03:04.6759364Z 
2023-01-11T21:03:04.6759436Z Generating XML reports...
2023-01-11T21:03:04.6759775Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205231.xml
2023-01-11T21:03:04.6760003Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6760284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6760407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6760690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6760823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6760828Z 
2023-01-11T21:03:04.6760899Z Running tests...
2023-01-11T21:03:04.6761095Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6761276Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2023-01-11T21:03:04.6761292Z 
2023-01-11T21:03:04.6761471Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6761543Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6761548Z 
2023-01-11T21:03:04.6761621Z OK (skipped=1)
2023-01-11T21:03:04.6761626Z 
2023-01-11T21:03:04.6761709Z Generating XML reports...
2023-01-11T21:03:04.6762046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205232.xml
2023-01-11T21:03:04.6762275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6762544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6762733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6763005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6763138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6763145Z 
2023-01-11T21:03:04.6763218Z Running tests...
2023-01-11T21:03:04.6763414Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6763604Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2023-01-11T21:03:04.6763610Z 
2023-01-11T21:03:04.6763803Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6763880Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6763885Z 
2023-01-11T21:03:04.6763957Z OK (skipped=1)
2023-01-11T21:03:04.6763961Z 
2023-01-11T21:03:04.6764049Z Generating XML reports...
2023-01-11T21:03:04.6764376Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205234.xml
2023-01-11T21:03:04.6764609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6764884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6765008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6765327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6765489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6765495Z 
2023-01-11T21:03:04.6765568Z Running tests...
2023-01-11T21:03:04.6765762Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6765946Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2023-01-11T21:03:04.6765968Z 
2023-01-11T21:03:04.6766151Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6766227Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6766233Z 
2023-01-11T21:03:04.6766305Z OK (skipped=1)
2023-01-11T21:03:04.6766310Z 
2023-01-11T21:03:04.6766394Z Generating XML reports...
2023-01-11T21:03:04.6766734Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205236.xml
2023-01-11T21:03:04.6766969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6767247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6767370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6767639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6767774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6767779Z 
2023-01-11T21:03:04.6767855Z Running tests...
2023-01-11T21:03:04.6768053Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6768238Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2023-01-11T21:03:04.6768244Z 
2023-01-11T21:03:04.6768439Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6768516Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6768522Z 
2023-01-11T21:03:04.6768593Z OK (skipped=1)
2023-01-11T21:03:04.6768597Z 
2023-01-11T21:03:04.6768684Z Generating XML reports...
2023-01-11T21:03:04.6769006Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205237.xml
2023-01-11T21:03:04.6769239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6769513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6769637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6769918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6770052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6770060Z 
2023-01-11T21:03:04.6770134Z Running tests...
2023-01-11T21:03:04.6770329Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6770517Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:03:04.6770536Z 
2023-01-11T21:03:04.6770715Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6770792Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6770797Z 
2023-01-11T21:03:04.6770869Z OK (skipped=1)
2023-01-11T21:03:04.6770874Z 
2023-01-11T21:03:04.6770957Z Generating XML reports...
2023-01-11T21:03:04.6771296Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205239.xml
2023-01-11T21:03:04.6771528Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6771807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6771959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6772261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6772400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6772405Z 
2023-01-11T21:03:04.6772478Z Running tests...
2023-01-11T21:03:04.6772674Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6772880Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:03:04.6772886Z 
2023-01-11T21:03:04.6773077Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6773152Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6773158Z 
2023-01-11T21:03:04.6773227Z OK (skipped=1)
2023-01-11T21:03:04.6773234Z 
2023-01-11T21:03:04.6773318Z Generating XML reports...
2023-01-11T21:03:04.6773643Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205241.xml
2023-01-11T21:03:04.6773874Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6774145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6774268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6774548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6774681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6774686Z 
2023-01-11T21:03:04.6774756Z Running tests...
2023-01-11T21:03:04.6774952Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6775168Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:03:04.6775174Z 
2023-01-11T21:03:04.6775354Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6775426Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6775432Z 
2023-01-11T21:03:04.6775502Z OK (skipped=1)
2023-01-11T21:03:04.6775507Z 
2023-01-11T21:03:04.6775588Z Generating XML reports...
2023-01-11T21:03:04.6775925Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205242.xml
2023-01-11T21:03:04.6776154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6776427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6776546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6776825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6776948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6776953Z 
2023-01-11T21:03:04.6777023Z Running tests...
2023-01-11T21:03:04.6777219Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6777432Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2023-01-11T21:03:04.6777438Z 
2023-01-11T21:03:04.6777631Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6777704Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6777710Z 
2023-01-11T21:03:04.6777779Z OK (skipped=1)
2023-01-11T21:03:04.6777784Z 
2023-01-11T21:03:04.6777866Z Generating XML reports...
2023-01-11T21:03:04.6778188Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205244.xml
2023-01-11T21:03:04.6778459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6778760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6778883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6779165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6779303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6779308Z 
2023-01-11T21:03:04.6779379Z Running tests...
2023-01-11T21:03:04.6779574Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6779794Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:03:04.6779801Z 
2023-01-11T21:03:04.6779980Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6780052Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6780057Z 
2023-01-11T21:03:04.6780128Z OK (skipped=1)
2023-01-11T21:03:04.6780134Z 
2023-01-11T21:03:04.6780217Z Generating XML reports...
2023-01-11T21:03:04.6780554Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205246.xml
2023-01-11T21:03:04.6780784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6781056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6781176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6781452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6781575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6781582Z 
2023-01-11T21:03:04.6781739Z Running tests...
2023-01-11T21:03:04.6781936Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6782154Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:03:04.6782159Z 
2023-01-11T21:03:04.6782352Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6782427Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6782432Z 
2023-01-11T21:03:04.6782504Z OK (skipped=1)
2023-01-11T21:03:04.6782509Z 
2023-01-11T21:03:04.6782593Z Generating XML reports...
2023-01-11T21:03:04.6782931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205247.xml
2023-01-11T21:03:04.6783148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6783425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6783551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6783833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6783968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6783973Z 
2023-01-11T21:03:04.6784046Z Running tests...
2023-01-11T21:03:04.6784243Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6784465Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2023-01-11T21:03:04.6784471Z 
2023-01-11T21:03:04.6784661Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6784723Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6784778Z 
2023-01-11T21:03:04.6784849Z OK (skipped=1)
2023-01-11T21:03:04.6784854Z 
2023-01-11T21:03:04.6784939Z Generating XML reports...
2023-01-11T21:03:04.6785315Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205249.xml
2023-01-11T21:03:04.6785549Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6785823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6785945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6786224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6786345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6786364Z 
2023-01-11T21:03:04.6786423Z Running tests...
2023-01-11T21:03:04.6786617Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6786831Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:03:04.6786837Z 
2023-01-11T21:03:04.6787027Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6787100Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6787107Z 
2023-01-11T21:03:04.6787179Z OK (skipped=1)
2023-01-11T21:03:04.6787184Z 
2023-01-11T21:03:04.6787265Z Generating XML reports...
2023-01-11T21:03:04.6787601Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205251.xml
2023-01-11T21:03:04.6787817Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6788088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6788213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6788496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6788632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6788637Z 
2023-01-11T21:03:04.6788708Z Running tests...
2023-01-11T21:03:04.6788901Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6789121Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2023-01-11T21:03:04.6789127Z 
2023-01-11T21:03:04.6789318Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6789379Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6789384Z 
2023-01-11T21:03:04.6789454Z OK (skipped=1)
2023-01-11T21:03:04.6789459Z 
2023-01-11T21:03:04.6789540Z Generating XML reports...
2023-01-11T21:03:04.6789939Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205253.xml
2023-01-11T21:03:04.6790202Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6790474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6790596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6790876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6791007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6791012Z 
2023-01-11T21:03:04.6791071Z Running tests...
2023-01-11T21:03:04.6791262Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6791469Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:03:04.6791512Z 
2023-01-11T21:03:04.6791706Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6791811Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6791816Z 
2023-01-11T21:03:04.6791889Z OK (skipped=1)
2023-01-11T21:03:04.6791894Z 
2023-01-11T21:03:04.6791978Z Generating XML reports...
2023-01-11T21:03:04.6792314Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205254.xml
2023-01-11T21:03:04.6792545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6792805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6792928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6793213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6793354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6793359Z 
2023-01-11T21:03:04.6793429Z Running tests...
2023-01-11T21:03:04.6793625Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6793843Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:03:04.6793848Z 
2023-01-11T21:03:04.6794038Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6794100Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6794116Z 
2023-01-11T21:03:04.6794174Z OK (skipped=1)
2023-01-11T21:03:04.6794179Z 
2023-01-11T21:03:04.6794260Z Generating XML reports...
2023-01-11T21:03:04.6794594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205256.xml
2023-01-11T21:03:04.6794822Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6795096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6795220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6795497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6795629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6795634Z 
2023-01-11T21:03:04.6795693Z Running tests...
2023-01-11T21:03:04.6795885Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6796098Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2023-01-11T21:03:04.6796103Z 
2023-01-11T21:03:04.6796293Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6796370Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6796375Z 
2023-01-11T21:03:04.6796444Z OK (skipped=1)
2023-01-11T21:03:04.6796449Z 
2023-01-11T21:03:04.6796533Z Generating XML reports...
2023-01-11T21:03:04.6796869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205258.xml
2023-01-11T21:03:04.6797097Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6797360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6797480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6797756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6797889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6797895Z 
2023-01-11T21:03:04.6798003Z Running tests...
2023-01-11T21:03:04.6798198Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6798480Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:03:04.6798486Z 
2023-01-11T21:03:04.6798679Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6798752Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6798757Z 
2023-01-11T21:03:04.6798815Z OK (skipped=1)
2023-01-11T21:03:04.6798820Z 
2023-01-11T21:03:04.6798902Z Generating XML reports...
2023-01-11T21:03:04.6799240Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205259.xml
2023-01-11T21:03:04.6799470Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6799741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6799868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6800151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6800284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6800290Z 
2023-01-11T21:03:04.6800360Z Running tests...
2023-01-11T21:03:04.6800540Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6800757Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:03:04.6800762Z 
2023-01-11T21:03:04.6800958Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6801031Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6801036Z 
2023-01-11T21:03:04.6801105Z OK (skipped=1)
2023-01-11T21:03:04.6801113Z 
2023-01-11T21:03:04.6801195Z Generating XML reports...
2023-01-11T21:03:04.6801529Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205301.xml
2023-01-11T21:03:04.6801760Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6802022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6802143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6802420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6802552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6802558Z 
2023-01-11T21:03:04.6802629Z Running tests...
2023-01-11T21:03:04.6802897Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6803127Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2023-01-11T21:03:04.6803133Z 
2023-01-11T21:03:04.6803330Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6803404Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6803409Z 
2023-01-11T21:03:04.6803466Z OK (skipped=1)
2023-01-11T21:03:04.6803484Z 
2023-01-11T21:03:04.6803556Z Generating XML reports...
2023-01-11T21:03:04.6803891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205303.xml
2023-01-11T21:03:04.6804122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6804394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6804518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6804846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6805011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6805017Z 
2023-01-11T21:03:04.6805088Z Running tests...
2023-01-11T21:03:04.6805268Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6805480Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:03:04.6805485Z 
2023-01-11T21:03:04.6805676Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6805751Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6805756Z 
2023-01-11T21:03:04.6805825Z OK (skipped=1)
2023-01-11T21:03:04.6805830Z 
2023-01-11T21:03:04.6805912Z Generating XML reports...
2023-01-11T21:03:04.6806249Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205304.xml
2023-01-11T21:03:04.6806482Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6806758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6806867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6807147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6807280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6807285Z 
2023-01-11T21:03:04.6807355Z Running tests...
2023-01-11T21:03:04.6807550Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6807770Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2023-01-11T21:03:04.6807777Z 
2023-01-11T21:03:04.6807970Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6808044Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6808050Z 
2023-01-11T21:03:04.6808121Z OK (skipped=1)
2023-01-11T21:03:04.6808126Z 
2023-01-11T21:03:04.6808196Z Generating XML reports...
2023-01-11T21:03:04.6808528Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205306.xml
2023-01-11T21:03:04.6808755Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6809028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6809149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6809428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6809562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6809570Z 
2023-01-11T21:03:04.6809640Z Running tests...
2023-01-11T21:03:04.6809824Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6810056Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7774
2023-01-11T21:03:04.6810213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7775
2023-01-11T21:03:04.6810364Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7776
2023-01-11T21:03:04.6810636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6810758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6811039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6811204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6811377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6811664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6811787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6812066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6812198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6812372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6812641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6812765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6813045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6813179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6813340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6813639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6813936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6814229Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6814393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6814558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6814720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6814825Z skip: Need at least 2 CUDA devices (2.184s)
2023-01-11T21:03:04.6814831Z 
2023-01-11T21:03:04.6815028Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6815090Z Ran 1 test in 2.184s
2023-01-11T21:03:04.6815095Z 
2023-01-11T21:03:04.6815166Z OK (skipped=1)
2023-01-11T21:03:04.6815171Z 
2023-01-11T21:03:04.6815257Z Generating XML reports...
2023-01-11T21:03:04.6815597Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205308.xml
2023-01-11T21:03:04.6815827Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6816099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6816225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6816508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6816629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6816648Z 
2023-01-11T21:03:04.6816708Z Running tests...
2023-01-11T21:03:04.6816903Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6817219Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s)
2023-01-11T21:03:04.6817225Z 
2023-01-11T21:03:04.6817420Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6817494Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6817500Z 
2023-01-11T21:03:04.6817571Z OK (skipped=1)
2023-01-11T21:03:04.6817576Z 
2023-01-11T21:03:04.6817696Z Generating XML reports...
2023-01-11T21:03:04.6818039Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205312.xml
2023-01-11T21:03:04.6818316Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6818591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6818714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6818995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6819131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6819137Z 
2023-01-11T21:03:04.6819209Z Running tests...
2023-01-11T21:03:04.6819407Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6819718Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s)
2023-01-11T21:03:04.6819726Z 
2023-01-11T21:03:04.6819923Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6819984Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6819990Z 
2023-01-11T21:03:04.6820061Z OK (skipped=1)
2023-01-11T21:03:04.6820065Z 
2023-01-11T21:03:04.6820150Z Generating XML reports...
2023-01-11T21:03:04.6820488Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205313.xml
2023-01-11T21:03:04.6820722Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6820995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6821121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6821402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6821525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6821546Z 
2023-01-11T21:03:04.6821608Z Running tests...
2023-01-11T21:03:04.6821909Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6822134Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7847
2023-01-11T21:03:04.6822293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7848
2023-01-11T21:03:04.6822447Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7849
2023-01-11T21:03:04.6822723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6822847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6823132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6823252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6823434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6823706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6823829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6824108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6824242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6824419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6824689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6824880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6825181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6825323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6825498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6825802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6826101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6826396Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6826565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6826732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6826893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6826949Z ok (3.485s)
2023-01-11T21:03:04.6826955Z 
2023-01-11T21:03:04.6827153Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6827228Z Ran 1 test in 3.486s
2023-01-11T21:03:04.6827233Z 
2023-01-11T21:03:04.6827296Z OK
2023-01-11T21:03:04.6827302Z 
2023-01-11T21:03:04.6827386Z Generating XML reports...
2023-01-11T21:03:04.6827730Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205315.xml
2023-01-11T21:03:04.6827965Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6828240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6828350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6828633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6828768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6828773Z 
2023-01-11T21:03:04.6828845Z Running tests...
2023-01-11T21:03:04.6829039Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6829265Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7900
2023-01-11T21:03:04.6829417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7901
2023-01-11T21:03:04.6829566Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7902
2023-01-11T21:03:04.6829840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6829951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6830231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6830367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6830541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6830810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6830930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6831206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6831370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6831533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6831827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6831949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6832226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6832358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6832530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6832830Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6833124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6833415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6833566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6833720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6833883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6833978Z skip: CUDA is not available. (2.280s)
2023-01-11T21:03:04.6833984Z 
2023-01-11T21:03:04.6834180Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6834255Z Ran 1 test in 2.281s
2023-01-11T21:03:04.6834261Z 
2023-01-11T21:03:04.6834330Z OK (skipped=1)
2023-01-11T21:03:04.6834336Z 
2023-01-11T21:03:04.6834419Z Generating XML reports...
2023-01-11T21:03:04.6834762Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205320.xml
2023-01-11T21:03:04.6834982Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6835256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6835378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6835655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6835787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6835793Z 
2023-01-11T21:03:04.6835864Z Running tests...
2023-01-11T21:03:04.6836058Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6836287Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7953
2023-01-11T21:03:04.6836432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7954
2023-01-11T21:03:04.6836585Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7955
2023-01-11T21:03:04.6836856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6836978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6837258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6837390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6837564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6837834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6837992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6838287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6838422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6838599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6838897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6839170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6839293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6839572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6839706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6839881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6840166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6840459Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6840620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6840783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6840941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6841113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6841289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6841464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6841758Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6842040Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6842332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6842401Z ok (3.473s)
2023-01-11T21:03:04.6842406Z 
2023-01-11T21:03:04.6842602Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6842739Z Ran 1 test in 3.473s
2023-01-11T21:03:04.6842745Z 
2023-01-11T21:03:04.6842811Z OK
2023-01-11T21:03:04.6842816Z 
2023-01-11T21:03:04.6842901Z Generating XML reports...
2023-01-11T21:03:04.6843245Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205325.xml
2023-01-11T21:03:04.6843465Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6843739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6843862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6844144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6844279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6844285Z 
2023-01-11T21:03:04.6844358Z Running tests...
2023-01-11T21:03:04.6844559Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6844833Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8015
2023-01-11T21:03:04.6845016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8016
2023-01-11T21:03:04.6845157Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8017
2023-01-11T21:03:04.6845432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6845552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6845833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6845968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6846143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6846417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6846541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6846821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6846940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6847114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6847383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6847506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6847786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6847921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6848094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6848394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6848680Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6848974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6849137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6849299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6849460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6849559Z skip: CUDA is not available. (2.173s)
2023-01-11T21:03:04.6849565Z 
2023-01-11T21:03:04.6849761Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6849836Z Ran 1 test in 2.174s
2023-01-11T21:03:04.6849842Z 
2023-01-11T21:03:04.6849912Z OK (skipped=1)
2023-01-11T21:03:04.6849917Z 
2023-01-11T21:03:04.6849988Z Generating XML reports...
2023-01-11T21:03:04.6850334Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205330.xml
2023-01-11T21:03:04.6850566Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6850838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6850961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6851238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6851405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6851411Z 
2023-01-11T21:03:04.6851482Z Running tests...
2023-01-11T21:03:04.6851706Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6851919Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8068
2023-01-11T21:03:04.6852072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8069
2023-01-11T21:03:04.6852223Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8070
2023-01-11T21:03:04.6852494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6852616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6852896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6853033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6853213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6853473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6853593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6853870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6854004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6854182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6854453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6854578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6854861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6854993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6855153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6855452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6855743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6856036Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6856196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6856362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6856525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6856698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6856867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6857025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6857316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6857605Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6857893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6857991Z ok (3.175s)
2023-01-11T21:03:04.6857997Z 
2023-01-11T21:03:04.6858228Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6858306Z Ran 1 test in 3.175s
2023-01-11T21:03:04.6858311Z 
2023-01-11T21:03:04.6858372Z OK
2023-01-11T21:03:04.6858379Z 
2023-01-11T21:03:04.6858461Z Generating XML reports...
2023-01-11T21:03:04.6858789Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205334.xml
2023-01-11T21:03:04.6859021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6859296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6859419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6859701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6859833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6859842Z 
2023-01-11T21:03:04.6859913Z Running tests...
2023-01-11T21:03:04.6860112Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6860331Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8127
2023-01-11T21:03:04.6860486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8128
2023-01-11T21:03:04.6860639Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8129
2023-01-11T21:03:04.6860910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6861036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6861321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6861457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6861631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6861993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6862104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6862386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6862520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6862695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6862995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6863267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6863389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6863669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6863803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6863962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6864256Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6864551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6864766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6864962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6865127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6865224Z skip: CUDA is not available. (2.173s)
2023-01-11T21:03:04.6865230Z 
2023-01-11T21:03:04.6865427Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6865489Z Ran 1 test in 2.173s
2023-01-11T21:03:04.6865508Z 
2023-01-11T21:03:04.6865566Z OK (skipped=1)
2023-01-11T21:03:04.6865571Z 
2023-01-11T21:03:04.6865655Z Generating XML reports...
2023-01-11T21:03:04.6866001Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205339.xml
2023-01-11T21:03:04.6866233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6866509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6866634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6866916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6867052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6867057Z 
2023-01-11T21:03:04.6867117Z Running tests...
2023-01-11T21:03:04.6867316Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6867555Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8180
2023-01-11T21:03:04.6867709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8181
2023-01-11T21:03:04.6867861Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8182
2023-01-11T21:03:04.6868137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6868260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6868539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6868671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6868833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6869102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6869224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6869501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6869637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6869813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6870083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6870204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6870471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6870608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6870781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6871081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6871422Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6871737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6871900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6872059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6872222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6872382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6872562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6872734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6873029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6873325Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6873613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6873684Z ok (3.374s)
2023-01-11T21:03:04.6873690Z 
2023-01-11T21:03:04.6873887Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6873963Z Ran 1 test in 3.375s
2023-01-11T21:03:04.6873968Z 
2023-01-11T21:03:04.6874017Z OK
2023-01-11T21:03:04.6874036Z 
2023-01-11T21:03:04.6874107Z Generating XML reports...
2023-01-11T21:03:04.6874449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205343.xml
2023-01-11T21:03:04.6874685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6874964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6875087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6875369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6875505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6875510Z 
2023-01-11T21:03:04.6875585Z Running tests...
2023-01-11T21:03:04.6875767Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6876076Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s)
2023-01-11T21:03:04.6876084Z 
2023-01-11T21:03:04.6876281Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6876357Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6876365Z 
2023-01-11T21:03:04.6876438Z OK (skipped=1)
2023-01-11T21:03:04.6876443Z 
2023-01-11T21:03:04.6876527Z Generating XML reports...
2023-01-11T21:03:04.6876868Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205348.xml
2023-01-11T21:03:04.6877099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6877375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6877486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6877769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6877937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6877942Z 
2023-01-11T21:03:04.6878013Z Running tests...
2023-01-11T21:03:04.6878238Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6878474Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8252
2023-01-11T21:03:04.6878628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8253
2023-01-11T21:03:04.6878781Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8254
2023-01-11T21:03:04.6879056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6879168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6879454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6879591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6879771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6880046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6880169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6880452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6880587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6880750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6881019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6881146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6881427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6881562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6881735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6942300Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6942802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6943112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6943282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6943461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6943619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6943799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.6943974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.6944149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.6944449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6944744Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6945033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.6945250Z ok (7.182s)
2023-01-11T21:03:04.6945258Z 
2023-01-11T21:03:04.6945511Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6945579Z Ran 1 test in 7.182s
2023-01-11T21:03:04.6945586Z 
2023-01-11T21:03:04.6945648Z OK
2023-01-11T21:03:04.6945653Z 
2023-01-11T21:03:04.6945739Z Generating XML reports...
2023-01-11T21:03:04.6946089Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205350.xml
2023-01-11T21:03:04.6946326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6946611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6946736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6947018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6947145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6947157Z 
2023-01-11T21:03:04.6947220Z Running tests...
2023-01-11T21:03:04.6947413Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6947644Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8311
2023-01-11T21:03:04.6947795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8312
2023-01-11T21:03:04.6947943Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8313
2023-01-11T21:03:04.6948210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6948328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6948606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6948732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6948902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6949170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6949293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6949574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6949706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6949883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6950152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6950276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6950546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6950678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6950849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6951147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6951442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6951730Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6951929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6952124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6952283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6952339Z ok (2.179s)
2023-01-11T21:03:04.6952345Z 
2023-01-11T21:03:04.6952538Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6952611Z Ran 1 test in 2.179s
2023-01-11T21:03:04.6952617Z 
2023-01-11T21:03:04.6952674Z OK
2023-01-11T21:03:04.6952680Z 
2023-01-11T21:03:04.6952760Z Generating XML reports...
2023-01-11T21:03:04.6953104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205359.xml
2023-01-11T21:03:04.6953332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6953611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6953723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6954009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6954144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6954150Z 
2023-01-11T21:03:04.6954221Z Running tests...
2023-01-11T21:03:04.6954417Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6954657Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8364
2023-01-11T21:03:04.6954813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8365
2023-01-11T21:03:04.6954965Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8366
2023-01-11T21:03:04.6955243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6955356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6955636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6955770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6955945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6956216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6956336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6956618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6956754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6956919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6957192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6957315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6957596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6957730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6957907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6958207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6958505Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6958861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6959016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6959180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6959342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6959411Z ok (2.180s)
2023-01-11T21:03:04.6959417Z 
2023-01-11T21:03:04.6959614Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6959689Z Ran 1 test in 2.180s
2023-01-11T21:03:04.6959695Z 
2023-01-11T21:03:04.6959756Z OK
2023-01-11T21:03:04.6959762Z 
2023-01-11T21:03:04.6959844Z Generating XML reports...
2023-01-11T21:03:04.6960176Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205403.xml
2023-01-11T21:03:04.6960410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6960688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6960811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6961094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6961231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6961236Z 
2023-01-11T21:03:04.6961310Z Running tests...
2023-01-11T21:03:04.6961504Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6961705Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:03:04.6961713Z 
2023-01-11T21:03:04.6961895Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6961970Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6961978Z 
2023-01-11T21:03:04.6962050Z OK (skipped=1)
2023-01-11T21:03:04.6962056Z 
2023-01-11T21:03:04.6962141Z Generating XML reports...
2023-01-11T21:03:04.6962481Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205407.xml
2023-01-11T21:03:04.6962783Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6963062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6963188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6963468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6963594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6963599Z 
2023-01-11T21:03:04.6963677Z Running tests...
2023-01-11T21:03:04.6963876Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6964063Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:03:04.6964069Z 
2023-01-11T21:03:04.6964264Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6964342Z Ran 1 test in 0.003s
2023-01-11T21:03:04.6964348Z 
2023-01-11T21:03:04.6964420Z OK (skipped=1)
2023-01-11T21:03:04.6964426Z 
2023-01-11T21:03:04.6964513Z Generating XML reports...
2023-01-11T21:03:04.6964855Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205408.xml
2023-01-11T21:03:04.6965074Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6965385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6965547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6965827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6965963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6965968Z 
2023-01-11T21:03:04.6966043Z Running tests...
2023-01-11T21:03:04.6966241Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6966441Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:03:04.6966446Z 
2023-01-11T21:03:04.6966641Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6966703Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6966711Z 
2023-01-11T21:03:04.6966786Z OK (skipped=1)
2023-01-11T21:03:04.6966791Z 
2023-01-11T21:03:04.6966878Z Generating XML reports...
2023-01-11T21:03:04.6967223Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205410.xml
2023-01-11T21:03:04.6967455Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6967733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6967858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6968137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6968260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6968277Z 
2023-01-11T21:03:04.6968337Z Running tests...
2023-01-11T21:03:04.6968535Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6968724Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:03:04.6968732Z 
2023-01-11T21:03:04.6968926Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6969002Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6969008Z 
2023-01-11T21:03:04.6969083Z OK (skipped=1)
2023-01-11T21:03:04.6969088Z 
2023-01-11T21:03:04.6969174Z Generating XML reports...
2023-01-11T21:03:04.6969519Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205412.xml
2023-01-11T21:03:04.6969738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6970017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6970142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6970428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6970565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6970570Z 
2023-01-11T21:03:04.6970642Z Running tests...
2023-01-11T21:03:04.6970838Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6971029Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:03:04.6971035Z 
2023-01-11T21:03:04.6971229Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6971293Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6971298Z 
2023-01-11T21:03:04.6971372Z OK (skipped=1)
2023-01-11T21:03:04.6971377Z 
2023-01-11T21:03:04.6971462Z Generating XML reports...
2023-01-11T21:03:04.6971800Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205414.xml
2023-01-11T21:03:04.6972067Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6972367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6972495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6972776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6972898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6972916Z 
2023-01-11T21:03:04.6972976Z Running tests...
2023-01-11T21:03:04.6973171Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6973371Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:03:04.6973379Z 
2023-01-11T21:03:04.6973571Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6973647Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6973652Z 
2023-01-11T21:03:04.6973726Z OK (skipped=1)
2023-01-11T21:03:04.6973731Z 
2023-01-11T21:03:04.6973815Z Generating XML reports...
2023-01-11T21:03:04.6974152Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205415.xml
2023-01-11T21:03:04.6974371Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6974647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6974771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6975051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6975188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6975194Z 
2023-01-11T21:03:04.6975267Z Running tests...
2023-01-11T21:03:04.6975463Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6975656Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:03:04.6975661Z 
2023-01-11T21:03:04.6975852Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6975914Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6975919Z 
2023-01-11T21:03:04.6975993Z OK (skipped=1)
2023-01-11T21:03:04.6975998Z 
2023-01-11T21:03:04.6976082Z Generating XML reports...
2023-01-11T21:03:04.6976416Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205417.xml
2023-01-11T21:03:04.6976640Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6976913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6977035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6977310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6977433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6977449Z 
2023-01-11T21:03:04.6977508Z Running tests...
2023-01-11T21:03:04.6977701Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6977893Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:03:04.6977898Z 
2023-01-11T21:03:04.6978091Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6978165Z Ran 1 test in 0.002s
2023-01-11T21:03:04.6978170Z 
2023-01-11T21:03:04.6978286Z OK (skipped=1)
2023-01-11T21:03:04.6978292Z 
2023-01-11T21:03:04.6978375Z Generating XML reports...
2023-01-11T21:03:04.6978734Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205419.xml
2023-01-11T21:03:04.6978956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.6979223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6979345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6979622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6979754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6979759Z 
2023-01-11T21:03:04.6979831Z Running tests...
2023-01-11T21:03:04.6980025Z ----------------------------------------------------------------------
2023-01-11T21:03:04.6980250Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8497
2023-01-11T21:03:04.6980405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8498
2023-01-11T21:03:04.6980547Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8499
2023-01-11T21:03:04.6980815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6980933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6981213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6981343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6981508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.6981887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6982012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6982292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6982424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6982602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.6982877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.6982996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.6983275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.6983399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.6983569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.6983864Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6984155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6984441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.6984599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.6984758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.6984915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.6985248Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6985523Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6985763Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6986000Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6986235Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6986469Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6986716Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6987142Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6987151Z 
2023-01-11T21:03:04.6987388Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6987619Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6987845Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6988247Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6988253Z 
2023-01-11T21:03:04.6988831Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6988841Z 
2023-01-11T21:03:04.6988851Z 
2023-01-11T21:03:04.6989094Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6989324Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6989555Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6989794Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6990026Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6990274Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6990514Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6990750Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6990994Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6991241Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6991472Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6991705Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6991935Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6992173Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6992770Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6992802Z 
2023-01-11T21:03:04.6992807Z 
2023-01-11T21:03:04.6993227Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6993232Z 
2023-01-11T21:03:04.6993462Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6993692Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6993921Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6994158Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6994564Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6994800Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6994806Z 
2023-01-11T21:03:04.6995050Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6995296Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6995525Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6995753Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6995983Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6996225Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6996627Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6996634Z 
2023-01-11T21:03:04.6996881Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6997296Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6997308Z 
2023-01-11T21:03:04.6997533Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6997766Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6997996Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6998236Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6998488Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6998729Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6999141Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.6999147Z 
2023-01-11T21:03:04.6999399Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.6999666Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.6999922Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7000162Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7000405Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7000659Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7000901Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7001145Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7001381Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7001626Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7001860Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7002085Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7002315Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7002549Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7002855Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7003099Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7003510Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7003518Z 
2023-01-11T21:03:04.7003765Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7003997Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7004230Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7004461Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7004691Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7004925Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7005170Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7005585Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7005590Z 
2023-01-11T21:03:04.7005835Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7006066Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7006297Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7006528Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7006762Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7007002Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7007272Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7007544Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7007784Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7008031Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7008263Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7008494Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7008724Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7009282Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7009290Z 
2023-01-11T21:03:04.7009295Z 
2023-01-11T21:03:04.7009873Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7009879Z 
2023-01-11T21:03:04.7009883Z 
2023-01-11T21:03:04.7010114Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7010346Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7010569Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7010809Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7011218Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7011224Z 
2023-01-11T21:03:04.7011462Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7011710Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7011956Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7012189Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7012419Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7012657Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7012895Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7013121Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7013370Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7013606Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7013852Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7014098Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7014333Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7014778Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7014785Z 
2023-01-11T21:03:04.7015022Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7015433Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7015439Z 
2023-01-11T21:03:04.7015676Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7015914Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7016161Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7016395Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7016623Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7016854Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7017089Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7017323Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7017559Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7017804Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7018045Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7018292Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7018525Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7018754Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7018984Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7019221Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7019464Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7019701Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7019937Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7020177Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7020424Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7020653Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7020882Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7021112Z STAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7021345Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7021590Z STAGE:2023-01-11 20:54:22 8498:8498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7022213Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7022220Z 
2023-01-11T21:03:04.7022639Z STAGE:2023-01-11 20:54:22 8499:8499 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:22 8497:8497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7022645Z 
2023-01-11T21:03:04.7022706Z ok (2.276s)
2023-01-11T21:03:04.7022711Z 
2023-01-11T21:03:04.7022897Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7022967Z Ran 1 test in 2.276s
2023-01-11T21:03:04.7022972Z 
2023-01-11T21:03:04.7023027Z OK
2023-01-11T21:03:04.7023032Z 
2023-01-11T21:03:04.7023109Z Generating XML reports...
2023-01-11T21:03:04.7023446Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205420.xml
2023-01-11T21:03:04.7023678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7023952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7024070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7024348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7024472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7024477Z 
2023-01-11T21:03:04.7024543Z Running tests...
2023-01-11T21:03:04.7024734Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7025414Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.269s)
2023-01-11T21:03:04.7025423Z 
2023-01-11T21:03:04.7025615Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7025684Z Ran 1 test in 0.269s
2023-01-11T21:03:04.7025689Z 
2023-01-11T21:03:04.7025754Z OK (skipped=1)
2023-01-11T21:03:04.7025759Z 
2023-01-11T21:03:04.7025837Z Generating XML reports...
2023-01-11T21:03:04.7026178Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205424.xml
2023-01-11T21:03:04.7026405Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7026666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7026785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7027061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7027193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7027198Z 
2023-01-11T21:03:04.7027263Z Running tests...
2023-01-11T21:03:04.7027450Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7027676Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8563
2023-01-11T21:03:04.7027825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8564
2023-01-11T21:03:04.7027966Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8565
2023-01-11T21:03:04.7028231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7028348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7028676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7028835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7029007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7029304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7029572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7029688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7029957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7030088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7030261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7030528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7030648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7030923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7031048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7031218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7031514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7031798Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7031964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7032122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7032282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7032451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7032622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7032787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7033078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7033364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7033644Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7033885Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7034279Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7034285Z 
2023-01-11T21:03:04.7034843Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7034849Z 
2023-01-11T21:03:04.7034854Z 
2023-01-11T21:03:04.7035497Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7035504Z 
2023-01-11T21:03:04.7035509Z 
2023-01-11T21:03:04.7035743Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7035976Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7036209Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7036610Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7036619Z 
2023-01-11T21:03:04.7036856Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7037275Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7037281Z 
2023-01-11T21:03:04.7037528Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7037760Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7037991Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7038216Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7038454Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7038707Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7038948Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7039182Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7039429Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7039675Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7039905Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7040134Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7040358Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7040598Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7041010Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7041252Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7041258Z 
2023-01-11T21:03:04.7041502Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7041751Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7041983Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7042215Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7042477Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7042810Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7043045Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7043293Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7043530Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7043775Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7044022Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7044255Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7044488Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7044720Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7044957Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7045350Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7045362Z 
2023-01-11T21:03:04.7045601Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7046019Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7046027Z 
2023-01-11T21:03:04.7046257Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7046490Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7046723Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7046961Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7047194Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7047602Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7047609Z 
2023-01-11T21:03:04.7047855Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7048105Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7048329Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7048719Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7048725Z 
2023-01-11T21:03:04.7049124Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7049130Z 
2023-01-11T21:03:04.7049369Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7049648Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7049923Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7050175Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7050405Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7050638Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7050869Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7051098Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7051330Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7051577Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7051827Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7052065Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7052313Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7052546Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7052778Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7053009Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7053239Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7053487Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7053725Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7053958Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7054202Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7054444Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7054677Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7054910Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7055143Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7055538Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7055551Z 
2023-01-11T21:03:04.7056120Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7056132Z 
2023-01-11T21:03:04.7056136Z 
2023-01-11T21:03:04.7056375Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7056607Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7056837Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7057099Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7057369Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7057616Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7057853Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7058086Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7058333Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7058570Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7058800Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7059032Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7059266Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7059503Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7059738Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7059972Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7060218Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7060464Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7060702Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7060942Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7061339Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7061344Z 
2023-01-11T21:03:04.7061585Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7062275Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7062281Z 
2023-01-11T21:03:04.7062285Z 
2023-01-11T21:03:04.7062710Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7062717Z 
2023-01-11T21:03:04.7062958Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7063347Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7063352Z 
2023-01-11T21:03:04.7063596Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7063840Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7064258Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7064313Z 
2023-01-11T21:03:04.7064569Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7064864Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7065093Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7065330Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7065568Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7065812Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7066053Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7066292Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7066551Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7066803Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7067054Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7067279Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7067668Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7067673Z 
2023-01-11T21:03:04.7067918Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7068329Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7068337Z 
2023-01-11T21:03:04.7068587Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7069009Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7069014Z 
2023-01-11T21:03:04.7069254Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7069494Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7069734Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7069981Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7070223Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7070451Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7070705Z STAGE:2023-01-11 20:54:28 8565:8565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7070958Z STAGE:2023-01-11 20:54:28 8563:8563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7071209Z STAGE:2023-01-11 20:54:28 8564:8564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7071277Z ok (2.275s)
2023-01-11T21:03:04.7071283Z 
2023-01-11T21:03:04.7071484Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7071561Z Ran 1 test in 2.275s
2023-01-11T21:03:04.7071595Z 
2023-01-11T21:03:04.7071656Z OK
2023-01-11T21:03:04.7071662Z 
2023-01-11T21:03:04.7071748Z Generating XML reports...
2023-01-11T21:03:04.7072105Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205426.xml
2023-01-11T21:03:04.7072339Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7072618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7072741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7073024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7073160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7073166Z 
2023-01-11T21:03:04.7073239Z Running tests...
2023-01-11T21:03:04.7073436Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7073668Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8628
2023-01-11T21:03:04.7073814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8629
2023-01-11T21:03:04.7073967Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8630
2023-01-11T21:03:04.7074243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7074367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7074652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7074788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7074968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7075244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7075358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7075637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7075772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7075951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7076226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7076350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7076630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7076769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7076946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7077235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7077534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7077826Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7077994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7078159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7078318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7078539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7078740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7078913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7079196Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7079486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7079733Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7079972Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7080223Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7080648Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7080655Z 
2023-01-11T21:03:04.7080909Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7081150Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7081392Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7081638Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7081880Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7082127Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7082382Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7082622Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7082955Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7083200Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7083451Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7083698Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7083935Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7084174Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7084413Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7084655Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7084906Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7085151Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7085403Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7085639Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7085877Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7086140Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7086420Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7086663Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7086913Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7087149Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7087386Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7087628Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7087867Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7088120Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7088358Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7088595Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7088831Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7089073Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7089310Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7089561Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7089813Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7090052Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7090293Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7090520Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7090792Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7091130Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7091384Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7091619Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7091917Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7092161Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7092412Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7092712Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7092950Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7093186Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7093477Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7093719Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7093969Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7094311Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7094681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7094999Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7095242Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7095468Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7095766Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7096023Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7096267Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7096579Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7096823Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7097062Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7097362Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7097604Z STAGE:2023-01-11 20:54:32 8630:8630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7097847Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7098224Z STAGE:2023-01-11 20:54:32 8629:8629 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7098297Z ok (2.378s)
2023-01-11T21:03:04.7098306Z 
2023-01-11T21:03:04.7098581Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7098685Z Ran 1 test in 2.379s
2023-01-11T21:03:04.7098692Z 
2023-01-11T21:03:04.7098789Z OK
2023-01-11T21:03:04.7098796Z 
2023-01-11T21:03:04.7098927Z Generating XML reports...
2023-01-11T21:03:04.7099475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205430.xml
2023-01-11T21:03:04.7099839Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7100312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7100514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7100953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7101149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7101157Z 
2023-01-11T21:03:04.7101258Z Running tests...
2023-01-11T21:03:04.7101543Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7102018Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8689
2023-01-11T21:03:04.7102166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8690
2023-01-11T21:03:04.7102323Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8691
2023-01-11T21:03:04.7102604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7102730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7103015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7103252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7103477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7103761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7103887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7104156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7104293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7104470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7104773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7105053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7105178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7105465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7105602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7105779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7106066Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7106365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7106531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7106701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7106867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7106964Z skip: CUDA is not available. (2.182s)
2023-01-11T21:03:04.7106970Z 
2023-01-11T21:03:04.7107168Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7107244Z Ran 1 test in 2.182s
2023-01-11T21:03:04.7107250Z 
2023-01-11T21:03:04.7107309Z OK (skipped=1)
2023-01-11T21:03:04.7107330Z 
2023-01-11T21:03:04.7107402Z Generating XML reports...
2023-01-11T21:03:04.7107745Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205435.xml
2023-01-11T21:03:04.7107981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7108259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7108389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7108673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7108810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7108815Z 
2023-01-11T21:03:04.7108888Z Running tests...
2023-01-11T21:03:04.7109072Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7109763Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.259s)
2023-01-11T21:03:04.7109782Z 
2023-01-11T21:03:04.7109966Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7110074Z Ran 1 test in 0.259s
2023-01-11T21:03:04.7110080Z 
2023-01-11T21:03:04.7110153Z OK (skipped=1)
2023-01-11T21:03:04.7110158Z 
2023-01-11T21:03:04.7110270Z Generating XML reports...
2023-01-11T21:03:04.7110617Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205439.xml
2023-01-11T21:03:04.7110851Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7111125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7111250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7111520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7111655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7111664Z 
2023-01-11T21:03:04.7111737Z Running tests...
2023-01-11T21:03:04.7111933Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7112317Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s)
2023-01-11T21:03:04.7112323Z 
2023-01-11T21:03:04.7112516Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7112591Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7112597Z 
2023-01-11T21:03:04.7112668Z OK (skipped=1)
2023-01-11T21:03:04.7112674Z 
2023-01-11T21:03:04.7112757Z Generating XML reports...
2023-01-11T21:03:04.7113085Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205441.xml
2023-01-11T21:03:04.7113317Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7113596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7113722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7114003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7114138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7114144Z 
2023-01-11T21:03:04.7114216Z Running tests...
2023-01-11T21:03:04.7114409Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7114793Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s)
2023-01-11T21:03:04.7114799Z 
2023-01-11T21:03:04.7114981Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7115058Z Ran 1 test in 0.001s
2023-01-11T21:03:04.7115064Z 
2023-01-11T21:03:04.7115136Z OK (skipped=1)
2023-01-11T21:03:04.7115141Z 
2023-01-11T21:03:04.7115227Z Generating XML reports...
2023-01-11T21:03:04.7115567Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205442.xml
2023-01-11T21:03:04.7115802Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7116080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7116204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7116484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7116606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7116611Z 
2023-01-11T21:03:04.7116684Z Running tests...
2023-01-11T21:03:04.7116911Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7117182Z   test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8772
2023-01-11T21:03:04.7117342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8773
2023-01-11T21:03:04.7117495Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8774
2023-01-11T21:03:04.7117771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7117894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7118164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7118300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7118482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7118759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7118884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7119164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7119299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7119475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7119751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7119862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7120141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7120279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7120458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7120760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7121057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7121348Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7121514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7121679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7121833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7121938Z skip: Need at least 2 CUDA devices (2.171s)
2023-01-11T21:03:04.7121944Z 
2023-01-11T21:03:04.7122145Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7122221Z Ran 1 test in 2.171s
2023-01-11T21:03:04.7122227Z 
2023-01-11T21:03:04.7122298Z OK (skipped=1)
2023-01-11T21:03:04.7122303Z 
2023-01-11T21:03:04.7122388Z Generating XML reports...
2023-01-11T21:03:04.7122808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205444.xml
2023-01-11T21:03:04.7123047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7123324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7123436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7123759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7123926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7123933Z 
2023-01-11T21:03:04.7124007Z Running tests...
2023-01-11T21:03:04.7124203Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7124471Z   test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8825
2023-01-11T21:03:04.7124627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8826
2023-01-11T21:03:04.7124781Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8827
2023-01-11T21:03:04.7125042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7125168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7125453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7125592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7125771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7126047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7126172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7126460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7126595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7126759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7127032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7127157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7127437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7127570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7127747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7128048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7128344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7128638Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7128792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7128957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7129118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7129228Z skip: Need at least 2 CUDA devices (2.182s)
2023-01-11T21:03:04.7129233Z 
2023-01-11T21:03:04.7129433Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7129508Z Ran 1 test in 2.182s
2023-01-11T21:03:04.7129513Z 
2023-01-11T21:03:04.7129585Z OK (skipped=1)
2023-01-11T21:03:04.7129590Z 
2023-01-11T21:03:04.7129675Z Generating XML reports...
2023-01-11T21:03:04.7130004Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205448.xml
2023-01-11T21:03:04.7130290Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7130597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7130721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7131003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7131138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7131143Z 
2023-01-11T21:03:04.7131217Z Running tests...
2023-01-11T21:03:04.7131412Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7131668Z   test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8878
2023-01-11T21:03:04.7131811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8879
2023-01-11T21:03:04.7131967Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8880
2023-01-11T21:03:04.7132242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7132366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7132651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7132789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7132969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7133244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7133369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7133639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7133775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7133952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7134226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7134350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7134631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7134767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7134946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7135235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7135535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7135828Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7135993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7136160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7136322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7136426Z skip: Need at least 2 CUDA devices (2.180s)
2023-01-11T21:03:04.7136432Z 
2023-01-11T21:03:04.7136629Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7136703Z Ran 1 test in 2.180s
2023-01-11T21:03:04.7136738Z 
2023-01-11T21:03:04.7136799Z OK (skipped=1)
2023-01-11T21:03:04.7136816Z 
2023-01-11T21:03:04.7136888Z Generating XML reports...
2023-01-11T21:03:04.7137260Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205452.xml
2023-01-11T21:03:04.7137497Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7137771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7137894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7138177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7138312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7138317Z 
2023-01-11T21:03:04.7138389Z Running tests...
2023-01-11T21:03:04.7138577Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7138815Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8931
2023-01-11T21:03:04.7138971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8932
2023-01-11T21:03:04.7139128Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8933
2023-01-11T21:03:04.7139402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7139527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7139810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7139945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7140109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7140384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7140509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7140791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7140926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7141104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7141381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7141506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7141909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7142036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7142216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7142517Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7142814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7143107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7143274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7143439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7143602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7143760Z skip: Need at least 2 CUDA devices (2.176s)
2023-01-11T21:03:04.7143765Z 
2023-01-11T21:03:04.7143988Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7144067Z Ran 1 test in 2.177s
2023-01-11T21:03:04.7144072Z 
2023-01-11T21:03:04.7144145Z OK (skipped=1)
2023-01-11T21:03:04.7144149Z 
2023-01-11T21:03:04.7144236Z Generating XML reports...
2023-01-11T21:03:04.7144583Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205456.xml
2023-01-11T21:03:04.7144818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7145097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7145221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7145508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7145630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7145638Z 
2023-01-11T21:03:04.7145711Z Running tests...
2023-01-11T21:03:04.7145910Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7146158Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8984
2023-01-11T21:03:04.7146313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8985
2023-01-11T21:03:04.7146466Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8986
2023-01-11T21:03:04.7146741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7146864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7147134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7147271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7147449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7147724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7147848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7148130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7148266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7148445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7148720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7148830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7149112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7149246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7149423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7149724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7150023Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7150316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7150517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7150709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7150860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7150964Z skip: Need at least 2 CUDA devices (2.280s)
2023-01-11T21:03:04.7150970Z 
2023-01-11T21:03:04.7151166Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7151241Z Ran 1 test in 2.280s
2023-01-11T21:03:04.7151247Z 
2023-01-11T21:03:04.7151318Z OK (skipped=1)
2023-01-11T21:03:04.7151324Z 
2023-01-11T21:03:04.7151408Z Generating XML reports...
2023-01-11T21:03:04.7151748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205500.xml
2023-01-11T21:03:04.7151980Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7152246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7152371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7152653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7152789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7152794Z 
2023-01-11T21:03:04.7152866Z Running tests...
2023-01-11T21:03:04.7153062Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7153760Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2023-01-11T21:03:04.7153769Z 
2023-01-11T21:03:04.7153964Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7154039Z Ran 1 test in 0.256s
2023-01-11T21:03:04.7154045Z 
2023-01-11T21:03:04.7154115Z OK (skipped=1)
2023-01-11T21:03:04.7154121Z 
2023-01-11T21:03:04.7154193Z Generating XML reports...
2023-01-11T21:03:04.7154535Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205504.xml
2023-01-11T21:03:04.7154766Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7155042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7155167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7155448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7155586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7155592Z 
2023-01-11T21:03:04.7155667Z Running tests...
2023-01-11T21:03:04.7155864Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7156555Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.260s)
2023-01-11T21:03:04.7156574Z 
2023-01-11T21:03:04.7156758Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7156833Z Ran 1 test in 0.260s
2023-01-11T21:03:04.7156838Z 
2023-01-11T21:03:04.7156944Z OK (skipped=1)
2023-01-11T21:03:04.7156949Z 
2023-01-11T21:03:04.7157033Z Generating XML reports...
2023-01-11T21:03:04.7157403Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205506.xml
2023-01-11T21:03:04.7157641Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7157916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7158039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7158320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7158443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7158449Z 
2023-01-11T21:03:04.7158520Z Running tests...
2023-01-11T21:03:04.7158718Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7159070Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.003s)
2023-01-11T21:03:04.7159078Z 
2023-01-11T21:03:04.7159273Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7159348Z Ran 1 test in 0.003s
2023-01-11T21:03:04.7159354Z 
2023-01-11T21:03:04.7159426Z OK (skipped=1)
2023-01-11T21:03:04.7159431Z 
2023-01-11T21:03:04.7159515Z Generating XML reports...
2023-01-11T21:03:04.7159855Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205508.xml
2023-01-11T21:03:04.7160073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7160349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7160474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7160759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7160896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7160901Z 
2023-01-11T21:03:04.7160975Z Running tests...
2023-01-11T21:03:04.7161171Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7161437Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9067
2023-01-11T21:03:04.7161594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9068
2023-01-11T21:03:04.7161736Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9069
2023-01-11T21:03:04.7162009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7162135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7162417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7162554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7162803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7163085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7163210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7163482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7163619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7163799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7164111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7164265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7164549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7164684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7164860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7165160Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7165445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7165740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7165910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7166076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7166240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7166345Z skip: Need at least 2 CUDA devices (2.176s)
2023-01-11T21:03:04.7166351Z 
2023-01-11T21:03:04.7166547Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7166623Z Ran 1 test in 2.177s
2023-01-11T21:03:04.7166628Z 
2023-01-11T21:03:04.7166699Z OK (skipped=1)
2023-01-11T21:03:04.7166705Z 
2023-01-11T21:03:04.7166777Z Generating XML reports...
2023-01-11T21:03:04.7167118Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205510.xml
2023-01-11T21:03:04.7167353Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7167630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7167756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7168038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7168175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7168180Z 
2023-01-11T21:03:04.7168253Z Running tests...
2023-01-11T21:03:04.7168435Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7168670Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9120
2023-01-11T21:03:04.7168825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9121
2023-01-11T21:03:04.7168981Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9122
2023-01-11T21:03:04.7169255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7169378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7169661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7169798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7169976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7170239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7170362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7170674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7170838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7171017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7171289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7171413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7171692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7171827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7171991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7172294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7172596Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7172890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7173056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7173221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7173382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7173486Z skip: Need at least 3 CUDA devices (2.184s)
2023-01-11T21:03:04.7173492Z 
2023-01-11T21:03:04.7173690Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7173756Z Ran 1 test in 2.185s
2023-01-11T21:03:04.7173761Z 
2023-01-11T21:03:04.7173832Z OK (skipped=1)
2023-01-11T21:03:04.7173837Z 
2023-01-11T21:03:04.7173921Z Generating XML reports...
2023-01-11T21:03:04.7174268Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205514.xml
2023-01-11T21:03:04.7174502Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7174776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7174899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7175180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7175302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7175319Z 
2023-01-11T21:03:04.7175379Z Running tests...
2023-01-11T21:03:04.7175575Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7175927Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.004s)
2023-01-11T21:03:04.7175933Z 
2023-01-11T21:03:04.7176129Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7176204Z Ran 1 test in 0.004s
2023-01-11T21:03:04.7176210Z 
2023-01-11T21:03:04.7176281Z OK (skipped=1)
2023-01-11T21:03:04.7176286Z 
2023-01-11T21:03:04.7176370Z Generating XML reports...
2023-01-11T21:03:04.7176710Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205518.xml
2023-01-11T21:03:04.7176932Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7177208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7177381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7177695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7177833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7177839Z 
2023-01-11T21:03:04.7177913Z Running tests...
2023-01-11T21:03:04.7178110Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7178450Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.003s)
2023-01-11T21:03:04.7178456Z 
2023-01-11T21:03:04.7178649Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7178710Z Ran 1 test in 0.003s
2023-01-11T21:03:04.7178716Z 
2023-01-11T21:03:04.7178788Z OK (skipped=1)
2023-01-11T21:03:04.7178792Z 
2023-01-11T21:03:04.7178880Z Generating XML reports...
2023-01-11T21:03:04.7179214Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205520.xml
2023-01-11T21:03:04.7179450Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7179726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7179850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7180134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7180270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7180275Z 
2023-01-11T21:03:04.7180335Z Running tests...
2023-01-11T21:03:04.7180529Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7180761Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9193
2023-01-11T21:03:04.7180923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9194
2023-01-11T21:03:04.7181078Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9195
2023-01-11T21:03:04.7181350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7181474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7181869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7181993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7182170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7182473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7182753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7182880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7183160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7183296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7183475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7183753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7183862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7184141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7184332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7184543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7184846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7185147Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7185314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7185479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7185644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7186327Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7187027Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7187703Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7188606Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.)
2023-01-11T21:03:04.7188782Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2023-01-11T21:03:04.7188942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.7189838Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.)
2023-01-11T21:03:04.7190012Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2023-01-11T21:03:04.7190183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.7191125Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.)
2023-01-11T21:03:04.7191332Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2023-01-11T21:03:04.7191504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.7192192Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7192883Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7193560Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7194227Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7194895Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7195556Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7196227Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7196909Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7197574Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7198269Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7198978Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7199638Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7200301Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7200959Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7201700Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:03:04.7201772Z ok (2.188s)
2023-01-11T21:03:04.7201778Z 
2023-01-11T21:03:04.7201979Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7202055Z Ran 1 test in 2.188s
2023-01-11T21:03:04.7202061Z 
2023-01-11T21:03:04.7202122Z OK
2023-01-11T21:03:04.7202127Z 
2023-01-11T21:03:04.7202211Z Generating XML reports...
2023-01-11T21:03:04.7202556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205522.xml
2023-01-11T21:03:04.7202862Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7203144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7203259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7203545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7203682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7203688Z 
2023-01-11T21:03:04.7203761Z Running tests...
2023-01-11T21:03:04.7203963Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7204270Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.004s)
2023-01-11T21:03:04.7204277Z 
2023-01-11T21:03:04.7204473Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7204549Z Ran 1 test in 0.004s
2023-01-11T21:03:04.7204555Z 
2023-01-11T21:03:04.7204614Z OK (skipped=1)
2023-01-11T21:03:04.7204633Z 
2023-01-11T21:03:04.7204783Z Generating XML reports...
2023-01-11T21:03:04.7205123Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205526.xml
2023-01-11T21:03:04.7205386Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7205663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7205789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7206074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7206211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7206217Z 
2023-01-11T21:03:04.7206289Z Running tests...
2023-01-11T21:03:04.7206472Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7206800Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s)
2023-01-11T21:03:04.7206809Z 
2023-01-11T21:03:04.7207007Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7207084Z Ran 1 test in 0.003s
2023-01-11T21:03:04.7207089Z 
2023-01-11T21:03:04.7207161Z OK (skipped=1)
2023-01-11T21:03:04.7207166Z 
2023-01-11T21:03:04.7207251Z Generating XML reports...
2023-01-11T21:03:04.7207594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205527.xml
2023-01-11T21:03:04.7207827Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7208103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7208213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7208494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7208632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7208639Z 
2023-01-11T21:03:04.7208712Z Running tests...
2023-01-11T21:03:04.7208909Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7209601Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.263s)
2023-01-11T21:03:04.7209608Z 
2023-01-11T21:03:04.7209803Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7209879Z Ran 1 test in 0.263s
2023-01-11T21:03:04.7209885Z 
2023-01-11T21:03:04.7209958Z OK (skipped=1)
2023-01-11T21:03:04.7209963Z 
2023-01-11T21:03:04.7210047Z Generating XML reports...
2023-01-11T21:03:04.7210378Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205529.xml
2023-01-11T21:03:04.7210612Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7210888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7211013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7211295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7211432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7211438Z 
2023-01-11T21:03:04.7211510Z Running tests...
2023-01-11T21:03:04.7211704Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7212456Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.256s)
2023-01-11T21:03:04.7212463Z 
2023-01-11T21:03:04.7212662Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7212725Z Ran 1 test in 0.256s
2023-01-11T21:03:04.7212730Z 
2023-01-11T21:03:04.7212801Z OK (skipped=1)
2023-01-11T21:03:04.7212806Z 
2023-01-11T21:03:04.7212889Z Generating XML reports...
2023-01-11T21:03:04.7213231Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205531.xml
2023-01-11T21:03:04.7213465Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7213743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7213870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7214153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7214276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7214293Z 
2023-01-11T21:03:04.7214353Z Running tests...
2023-01-11T21:03:04.7214547Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7214801Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9286
2023-01-11T21:03:04.7214959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9287
2023-01-11T21:03:04.7215113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9288
2023-01-11T21:03:04.7215391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7215516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7215797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7215920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7216100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7216373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7216497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7216776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7216914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7217095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7217368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7217491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7217759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7217893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7218071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7218374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7218705Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7219029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7219199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7219367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7219531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7219623Z skip: Need at least 3 CUDA devices (2.175s)
2023-01-11T21:03:04.7219629Z 
2023-01-11T21:03:04.7219825Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7219900Z Ran 1 test in 2.175s
2023-01-11T21:03:04.7219906Z 
2023-01-11T21:03:04.7219978Z OK (skipped=1)
2023-01-11T21:03:04.7219986Z 
2023-01-11T21:03:04.7220070Z Generating XML reports...
2023-01-11T21:03:04.7220414Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205533.xml
2023-01-11T21:03:04.7220649Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7220923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7221034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7221315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7221452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7221457Z 
2023-01-11T21:03:04.7221530Z Running tests...
2023-01-11T21:03:04.7221833Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7222080Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9339
2023-01-11T21:03:04.7222243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9340
2023-01-11T21:03:04.7222400Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9341
2023-01-11T21:03:04.7222675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7222786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7223067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7223204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7223383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7223659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7223783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7224067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7224204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7224369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7224643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7224768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7225051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7225186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7225432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7225768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7226069Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7226362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7226515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7226682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7226844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7226954Z skip: Need at least 3 CUDA devices (2.280s)
2023-01-11T21:03:04.7226960Z 
2023-01-11T21:03:04.7227158Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7227237Z Ran 1 test in 2.280s
2023-01-11T21:03:04.7227243Z 
2023-01-11T21:03:04.7227316Z OK (skipped=1)
2023-01-11T21:03:04.7227322Z 
2023-01-11T21:03:04.7227408Z Generating XML reports...
2023-01-11T21:03:04.7227751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205537.xml
2023-01-11T21:03:04.7227973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7228249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7228373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7228655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7228792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7228799Z 
2023-01-11T21:03:04.7228873Z Running tests...
2023-01-11T21:03:04.7229072Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7229772Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.273s)
2023-01-11T21:03:04.7229780Z 
2023-01-11T21:03:04.7229974Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7230049Z Ran 1 test in 0.273s
2023-01-11T21:03:04.7230055Z 
2023-01-11T21:03:04.7230113Z OK (skipped=1)
2023-01-11T21:03:04.7230118Z 
2023-01-11T21:03:04.7230205Z Generating XML reports...
2023-01-11T21:03:04.7230547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205541.xml
2023-01-11T21:03:04.7230783Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7231058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7231183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7231464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7231602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7231607Z 
2023-01-11T21:03:04.7231680Z Running tests...
2023-01-11T21:03:04.7231863Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7232100Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9402
2023-01-11T21:03:04.7232322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9403
2023-01-11T21:03:04.7232478Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9404
2023-01-11T21:03:04.7232754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7232879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7233162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7233299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7233464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7233736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7233864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7234146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7234283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7234460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7234732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7234855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7235135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7235256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7235433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7235736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7236035Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7236328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7236495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7236658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7236824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7236927Z skip: Need at least 3 CUDA devices (2.276s)
2023-01-11T21:03:04.7236935Z 
2023-01-11T21:03:04.7237120Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7237198Z Ran 1 test in 2.276s
2023-01-11T21:03:04.7237206Z 
2023-01-11T21:03:04.7237282Z OK (skipped=1)
2023-01-11T21:03:04.7237287Z 
2023-01-11T21:03:04.7237373Z Generating XML reports...
2023-01-11T21:03:04.7237717Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205543.xml
2023-01-11T21:03:04.7237949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7238224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7238348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7238615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7238781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7238787Z 
2023-01-11T21:03:04.7238861Z Running tests...
2023-01-11T21:03:04.7239089Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7239366Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9455
2023-01-11T21:03:04.7239524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9456
2023-01-11T21:03:04.7239676Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9457
2023-01-11T21:03:04.7239948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7240072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7240338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7240475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7240652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7240925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7241048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7241331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7241467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7241642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7241913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7242027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7242310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7242444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7242620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7242987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7243283Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7243579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7243746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7243912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7244065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7244170Z skip: Need at least 2 CUDA devices (2.178s)
2023-01-11T21:03:04.7244176Z 
2023-01-11T21:03:04.7244373Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7244449Z Ran 1 test in 2.178s
2023-01-11T21:03:04.7244455Z 
2023-01-11T21:03:04.7244527Z OK (skipped=1)
2023-01-11T21:03:04.7244533Z 
2023-01-11T21:03:04.7244619Z Generating XML reports...
2023-01-11T21:03:04.7244961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205547.xml
2023-01-11T21:03:04.7245195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7245458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7245622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7245935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7246075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7246081Z 
2023-01-11T21:03:04.7246155Z Running tests...
2023-01-11T21:03:04.7246356Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7246628Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9508
2023-01-11T21:03:04.7246786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9509
2023-01-11T21:03:04.7246940Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9510
2023-01-11T21:03:04.7247202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7247330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7247611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7247747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7247926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7248200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7248326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7248606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7248745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7248910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7249184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7249309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7249590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7249726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7249904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7250204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7250505Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7250789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7250954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7251118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7251281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7251390Z skip: Need at least 2 CUDA devices (2.178s)
2023-01-11T21:03:04.7251395Z 
2023-01-11T21:03:04.7251593Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7251669Z Ran 1 test in 2.178s
2023-01-11T21:03:04.7251674Z 
2023-01-11T21:03:04.7251748Z OK (skipped=1)
2023-01-11T21:03:04.7251753Z 
2023-01-11T21:03:04.7251871Z Generating XML reports...
2023-01-11T21:03:04.7252202Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205551.xml
2023-01-11T21:03:04.7252463Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7252741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7252865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7253148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7253283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7253290Z 
2023-01-11T21:03:04.7253362Z Running tests...
2023-01-11T21:03:04.7253559Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7253878Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9561
2023-01-11T21:03:04.7254027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9562
2023-01-11T21:03:04.7254181Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9563
2023-01-11T21:03:04.7254457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7254583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7254867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7255003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7255180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7255455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7255582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7255850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7255985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7256161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7256432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7256556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7256837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7256974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7257150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7257439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7257735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7258029Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7258194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7258358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7258520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7258655Z skip: Need at least 2 CUDA devices (2.185s)
2023-01-11T21:03:04.7258661Z 
2023-01-11T21:03:04.7258859Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7258968Z Ran 1 test in 2.185s
2023-01-11T21:03:04.7258974Z 
2023-01-11T21:03:04.7259035Z OK (skipped=1)
2023-01-11T21:03:04.7259052Z 
2023-01-11T21:03:04.7259124Z Generating XML reports...
2023-01-11T21:03:04.7259464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205555.xml
2023-01-11T21:03:04.7259699Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7259974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7260099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7260380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7260519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7260525Z 
2023-01-11T21:03:04.7260600Z Running tests...
2023-01-11T21:03:04.7260785Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7261100Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9614
2023-01-11T21:03:04.7261256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9615
2023-01-11T21:03:04.7261409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9616
2023-01-11T21:03:04.7261792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7261917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7262204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7262344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7262521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7262781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7262905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7263184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7263320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7263497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7263773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7263899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7264180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7264301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7264478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7264779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7265080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7265370Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7265589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7265792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7265958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7266063Z skip: Need at least 2 CUDA devices (2.178s)
2023-01-11T21:03:04.7266069Z 
2023-01-11T21:03:04.7266254Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7266331Z Ran 1 test in 2.178s
2023-01-11T21:03:04.7266336Z 
2023-01-11T21:03:04.7266411Z OK (skipped=1)
2023-01-11T21:03:04.7266416Z 
2023-01-11T21:03:04.7266503Z Generating XML reports...
2023-01-11T21:03:04.7266847Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205559.xml
2023-01-11T21:03:04.7267082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7267362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7267487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7267770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7267892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7267897Z 
2023-01-11T21:03:04.7267972Z Running tests...
2023-01-11T21:03:04.7268167Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7268480Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9667
2023-01-11T21:03:04.7268639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9668
2023-01-11T21:03:04.7268793Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9669
2023-01-11T21:03:04.7269068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7269191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7269470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7269592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7269771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7270047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7270172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7270455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7270593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7270772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7271043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7271154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7271435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7271569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7271744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7272091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7272416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7272713Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7272881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7273047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7273199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7273303Z skip: Need at least 2 CUDA devices (2.172s)
2023-01-11T21:03:04.7273309Z 
2023-01-11T21:03:04.7273506Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7273583Z Ran 1 test in 2.172s
2023-01-11T21:03:04.7273589Z 
2023-01-11T21:03:04.7273661Z OK (skipped=1)
2023-01-11T21:03:04.7273667Z 
2023-01-11T21:03:04.7273752Z Generating XML reports...
2023-01-11T21:03:04.7274096Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205603.xml
2023-01-11T21:03:04.7274334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7274613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7274726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7275011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7275148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7275154Z 
2023-01-11T21:03:04.7275231Z Running tests...
2023-01-11T21:03:04.7275429Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7275748Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9720
2023-01-11T21:03:04.7275906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9721
2023-01-11T21:03:04.7276058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9722
2023-01-11T21:03:04.7276332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7276442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7276724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7276860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7277042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7277318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7277442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7277723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7277857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7278033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7278292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7278416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7278731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7278896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7279075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7279374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7279671Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7279961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7280128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7280281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7280450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7280559Z skip: Need at least 2 CUDA devices (2.184s)
2023-01-11T21:03:04.7280564Z 
2023-01-11T21:03:04.7280762Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7280836Z Ran 1 test in 2.185s
2023-01-11T21:03:04.7280842Z 
2023-01-11T21:03:04.7280914Z OK (skipped=1)
2023-01-11T21:03:04.7280920Z 
2023-01-11T21:03:04.7281004Z Generating XML reports...
2023-01-11T21:03:04.7281349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205607.xml
2023-01-11T21:03:04.7281569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7281844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7281971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7282256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7282391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7282396Z 
2023-01-11T21:03:04.7282470Z Running tests...
2023-01-11T21:03:04.7282731Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7283052Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9773
2023-01-11T21:03:04.7283207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9774
2023-01-11T21:03:04.7283348Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9775
2023-01-11T21:03:04.7283624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7283751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7284036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7284173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7284353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7284625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7284751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7285033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7285154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7285370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7285678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7285804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7286084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7286218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7286395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7286693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7286979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7287276Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7287443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7287607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7287770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7287874Z skip: Need at least 2 CUDA devices (2.172s)
2023-01-11T21:03:04.7287880Z 
2023-01-11T21:03:04.7288077Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7288153Z Ran 1 test in 2.173s
2023-01-11T21:03:04.7288159Z 
2023-01-11T21:03:04.7288231Z OK (skipped=1)
2023-01-11T21:03:04.7288237Z 
2023-01-11T21:03:04.7288309Z Generating XML reports...
2023-01-11T21:03:04.7288655Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205611.xml
2023-01-11T21:03:04.7288890Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7289165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7289291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7289571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7289706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7289711Z 
2023-01-11T21:03:04.7289787Z Running tests...
2023-01-11T21:03:04.7289984Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7290283Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9826
2023-01-11T21:03:04.7290444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9827
2023-01-11T21:03:04.7290597Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9828
2023-01-11T21:03:04.7290870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7290993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7291277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7291412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7291588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7291860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7292006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7292319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7292457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7292635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7292909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7293032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7293311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7293445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7293610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7293912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7294211Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7294507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7294674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7294838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7295003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7295107Z skip: Need at least 2 CUDA devices (2.174s)
2023-01-11T21:03:04.7295115Z 
2023-01-11T21:03:04.7295310Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7295372Z Ran 1 test in 2.174s
2023-01-11T21:03:04.7295391Z 
2023-01-11T21:03:04.7295452Z OK (skipped=1)
2023-01-11T21:03:04.7295457Z 
2023-01-11T21:03:04.7295540Z Generating XML reports...
2023-01-11T21:03:04.7295882Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205615.xml
2023-01-11T21:03:04.7296115Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7296391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7296515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7296799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7296937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7296942Z 
2023-01-11T21:03:04.7297002Z Running tests...
2023-01-11T21:03:04.7297201Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7297516Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9879
2023-01-11T21:03:04.7297675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9880
2023-01-11T21:03:04.7297827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9881
2023-01-11T21:03:04.7298099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7298223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7298502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7298672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7298865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7299139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7299262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7299543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7299676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7299852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7300124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7300251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7300519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7300654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7300829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7301130Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7301429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7301834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7302006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7302176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7302343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7302434Z skip: Need at least 2 CUDA devices (2.176s)
2023-01-11T21:03:04.7302453Z 
2023-01-11T21:03:04.7302638Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7302714Z Ran 1 test in 2.177s
2023-01-11T21:03:04.7302720Z 
2023-01-11T21:03:04.7302793Z OK (skipped=1)
2023-01-11T21:03:04.7302798Z 
2023-01-11T21:03:04.7302885Z Generating XML reports...
2023-01-11T21:03:04.7303231Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205619.xml
2023-01-11T21:03:04.7303466Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7303744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7303870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7304139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7304276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7304281Z 
2023-01-11T21:03:04.7304356Z Running tests...
2023-01-11T21:03:04.7304551Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7304865Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9932
2023-01-11T21:03:04.7305020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9933
2023-01-11T21:03:04.7305225Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9934
2023-01-11T21:03:04.7305577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7305704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7305976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7306113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7306290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7306564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7306688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7306972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7307112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7307292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7307564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7307673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7307957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7308093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7308268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7308568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7308871Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7309166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7309332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7309497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7309648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7309753Z skip: Need at least 2 CUDA devices (2.181s)
2023-01-11T21:03:04.7309758Z 
2023-01-11T21:03:04.7309954Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7310033Z Ran 1 test in 2.181s
2023-01-11T21:03:04.7310041Z 
2023-01-11T21:03:04.7310113Z OK (skipped=1)
2023-01-11T21:03:04.7310118Z 
2023-01-11T21:03:04.7310203Z Generating XML reports...
2023-01-11T21:03:04.7310547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205623.xml
2023-01-11T21:03:04.7310781Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7311042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7311167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7311446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7311583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7311588Z 
2023-01-11T21:03:04.7311663Z Running tests...
2023-01-11T21:03:04.7311861Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7312189Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9985
2023-01-11T21:03:04.7312348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9986
2023-01-11T21:03:04.7312501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9987
2023-01-11T21:03:04.7312763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7312887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7313167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7313304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7313481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7313755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7313880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7314161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7314285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7314460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7314733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7314858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7315138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7315276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7315455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7315755Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7316053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7316332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7316496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7316663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7316825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7316932Z skip: Need at least 2 CUDA devices (2.173s)
2023-01-11T21:03:04.7316938Z 
2023-01-11T21:03:04.7317140Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7317215Z Ran 1 test in 2.174s
2023-01-11T21:03:04.7317221Z 
2023-01-11T21:03:04.7317293Z OK (skipped=1)
2023-01-11T21:03:04.7317298Z 
2023-01-11T21:03:04.7317383Z Generating XML reports...
2023-01-11T21:03:04.7317711Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205627.xml
2023-01-11T21:03:04.7317945Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7318220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7318345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7318629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7318819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7318853Z 
2023-01-11T21:03:04.7318929Z Running tests...
2023-01-11T21:03:04.7319127Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7319400Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10038
2023-01-11T21:03:04.7319545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10039
2023-01-11T21:03:04.7319702Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10040
2023-01-11T21:03:04.7319972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7320097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7320380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7320518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7320695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7320969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7321080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7321361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7321497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7321674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7321951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7322078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7322362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7322500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7322747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7323041Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7323340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7323637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7323809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7323979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7324143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7324249Z skip: Need at least 2 CUDA devices (2.181s)
2023-01-11T21:03:04.7324254Z 
2023-01-11T21:03:04.7324454Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7324530Z Ran 1 test in 2.182s
2023-01-11T21:03:04.7324535Z 
2023-01-11T21:03:04.7324594Z OK (skipped=1)
2023-01-11T21:03:04.7324598Z 
2023-01-11T21:03:04.7324684Z Generating XML reports...
2023-01-11T21:03:04.7325028Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205631.xml
2023-01-11T21:03:04.7325264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7325612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7325738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7326021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7326156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7326162Z 
2023-01-11T21:03:04.7326233Z Running tests...
2023-01-11T21:03:04.7326419Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7326743Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s)
2023-01-11T21:03:04.7326749Z 
2023-01-11T21:03:04.7326946Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7327024Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7327029Z 
2023-01-11T21:03:04.7327101Z OK (skipped=1)
2023-01-11T21:03:04.7327106Z 
2023-01-11T21:03:04.7327192Z Generating XML reports...
2023-01-11T21:03:04.7327532Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205635.xml
2023-01-11T21:03:04.7327768Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7328032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7328156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7328438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7328573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7328581Z 
2023-01-11T21:03:04.7328658Z Running tests...
2023-01-11T21:03:04.7328855Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7329087Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10101
2023-01-11T21:03:04.7329249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10102
2023-01-11T21:03:04.7329405Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10103
2023-01-11T21:03:04.7329669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7329793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7330074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7330209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7330389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7330664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7330787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7331066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7331204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7331368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7331642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7331764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7332051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7332220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7332424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7332725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7333021Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7333313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7333467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7333631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7333795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7333901Z skip: Need at least 2 CUDA devices (2.177s)
2023-01-11T21:03:04.7333909Z 
2023-01-11T21:03:04.7334108Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7334183Z Ran 1 test in 2.177s
2023-01-11T21:03:04.7334188Z 
2023-01-11T21:03:04.7334260Z OK (skipped=1)
2023-01-11T21:03:04.7334265Z 
2023-01-11T21:03:04.7334349Z Generating XML reports...
2023-01-11T21:03:04.7334679Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205637.xml
2023-01-11T21:03:04.7334914Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7335189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7335313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7335593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7335730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7335735Z 
2023-01-11T21:03:04.7335808Z Running tests...
2023-01-11T21:03:04.7336005Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7336246Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10154
2023-01-11T21:03:04.7336391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10155
2023-01-11T21:03:04.7336546Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10156
2023-01-11T21:03:04.7336818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7336945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7337229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7337364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7337540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7337814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7337924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7338204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7338339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7338520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7338824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7338977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7339259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7339393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7339567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7339853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7340152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7340444Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7340616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7340785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7340948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7341055Z skip: Need at least 2 CUDA devices (2.178s)
2023-01-11T21:03:04.7341060Z 
2023-01-11T21:03:04.7341258Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7341333Z Ran 1 test in 2.178s
2023-01-11T21:03:04.7341339Z 
2023-01-11T21:03:04.7341398Z OK (skipped=1)
2023-01-11T21:03:04.7341403Z 
2023-01-11T21:03:04.7341488Z Generating XML reports...
2023-01-11T21:03:04.7341923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205641.xml
2023-01-11T21:03:04.7342161Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7342440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7342564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7342847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7342986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7342992Z 
2023-01-11T21:03:04.7343067Z Running tests...
2023-01-11T21:03:04.7343248Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7343486Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10207
2023-01-11T21:03:04.7343644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10208
2023-01-11T21:03:04.7343801Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10209
2023-01-11T21:03:04.7344079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7344203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7344486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7344623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7344787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7345061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7345185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7345514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7345687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7345869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7346145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7346270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7346552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7346674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7346850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7347151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7347455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7347748Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7347916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7348080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7348240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7348416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.7348584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.7348754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:03:04.7348808Z ok (2.280s)
2023-01-11T21:03:04.7348815Z 
2023-01-11T21:03:04.7349017Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7349091Z Ran 1 test in 2.280s
2023-01-11T21:03:04.7349097Z 
2023-01-11T21:03:04.7349158Z OK
2023-01-11T21:03:04.7349164Z 
2023-01-11T21:03:04.7349248Z Generating XML reports...
2023-01-11T21:03:04.7349592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205645.xml
2023-01-11T21:03:04.7349826Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7350102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7350228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7350498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7350638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7350646Z 
2023-01-11T21:03:04.7350720Z Running tests...
2023-01-11T21:03:04.7350918Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7351155Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10269
2023-01-11T21:03:04.7351313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10270
2023-01-11T21:03:04.7351470Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10271
2023-01-11T21:03:04.7351743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7351855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7352170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7352336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7352516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7352788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7352914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7353200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7353336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7353518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7353779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7353908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7354193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7354329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7354505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7354804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7355105Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7355401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7355573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7355729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7355894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7355992Z skip: CUDA is not available. (2.188s)
2023-01-11T21:03:04.7355997Z 
2023-01-11T21:03:04.7356193Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7356269Z Ran 1 test in 2.188s
2023-01-11T21:03:04.7356275Z 
2023-01-11T21:03:04.7356347Z OK (skipped=1)
2023-01-11T21:03:04.7356352Z 
2023-01-11T21:03:04.7356437Z Generating XML reports...
2023-01-11T21:03:04.7356780Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205649.xml
2023-01-11T21:03:04.7357004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7357285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7357413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7357697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7357835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7357840Z 
2023-01-11T21:03:04.7357915Z Running tests...
2023-01-11T21:03:04.7358114Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7358464Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s)
2023-01-11T21:03:04.7358470Z 
2023-01-11T21:03:04.7358663Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7358759Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7358775Z 
2023-01-11T21:03:04.7358835Z OK (skipped=1)
2023-01-11T21:03:04.7358840Z 
2023-01-11T21:03:04.7358927Z Generating XML reports...
2023-01-11T21:03:04.7359298Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205653.xml
2023-01-11T21:03:04.7359535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7359813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7359940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7360223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7360361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7360366Z 
2023-01-11T21:03:04.7360430Z Running tests...
2023-01-11T21:03:04.7360626Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7360970Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s)
2023-01-11T21:03:04.7360976Z 
2023-01-11T21:03:04.7361166Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7361242Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7361247Z 
2023-01-11T21:03:04.7361319Z OK (skipped=1)
2023-01-11T21:03:04.7361325Z 
2023-01-11T21:03:04.7361409Z Generating XML reports...
2023-01-11T21:03:04.7361746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205655.xml
2023-01-11T21:03:04.7361976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7362238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7362363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7362729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7362873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7362878Z 
2023-01-11T21:03:04.7362956Z Running tests...
2023-01-11T21:03:04.7363155Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7363528Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s)
2023-01-11T21:03:04.7363534Z 
2023-01-11T21:03:04.7363731Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7363807Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7363813Z 
2023-01-11T21:03:04.7363872Z OK (skipped=1)
2023-01-11T21:03:04.7363879Z 
2023-01-11T21:03:04.7363966Z Generating XML reports...
2023-01-11T21:03:04.7364306Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205657.xml
2023-01-11T21:03:04.7364537Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7364813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7364938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7365220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7365359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7365364Z 
2023-01-11T21:03:04.7365440Z Running tests...
2023-01-11T21:03:04.7365621Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7366025Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s)
2023-01-11T21:03:04.7366030Z 
2023-01-11T21:03:04.7366250Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7366328Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7366333Z 
2023-01-11T21:03:04.7366404Z OK (skipped=1)
2023-01-11T21:03:04.7366410Z 
2023-01-11T21:03:04.7366494Z Generating XML reports...
2023-01-11T21:03:04.7366834Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205659.xml
2023-01-11T21:03:04.7367068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7367340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7367451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7367734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7367871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7367877Z 
2023-01-11T21:03:04.7367950Z Running tests...
2023-01-11T21:03:04.7368147Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7368459Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s)
2023-01-11T21:03:04.7368465Z 
2023-01-11T21:03:04.7368660Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7368737Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7368742Z 
2023-01-11T21:03:04.7368799Z OK (skipped=1)
2023-01-11T21:03:04.7368817Z 
2023-01-11T21:03:04.7368889Z Generating XML reports...
2023-01-11T21:03:04.7369224Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205700.xml
2023-01-11T21:03:04.7369458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7369734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7369858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7370138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7370273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7370278Z 
2023-01-11T21:03:04.7370352Z Running tests...
2023-01-11T21:03:04.7370533Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7370766Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10372
2023-01-11T21:03:04.7370924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10373
2023-01-11T21:03:04.7371082Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10374
2023-01-11T21:03:04.7371356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7371479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7371759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7371894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7372071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7372332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7372488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7372767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7372931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7373110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7373381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7373505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7373782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7373902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7374097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7374398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7374696Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7374991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7375160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7375323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7375485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7375588Z skip: Need at least 2 CUDA devices (2.178s)
2023-01-11T21:03:04.7375594Z 
2023-01-11T21:03:04.7375777Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7375855Z Ran 1 test in 2.178s
2023-01-11T21:03:04.7375860Z 
2023-01-11T21:03:04.7375931Z OK (skipped=1)
2023-01-11T21:03:04.7375936Z 
2023-01-11T21:03:04.7376023Z Generating XML reports...
2023-01-11T21:03:04.7376366Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205702.xml
2023-01-11T21:03:04.7376599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7376876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7377000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7377283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7377405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7377413Z 
2023-01-11T21:03:04.7377485Z Running tests...
2023-01-11T21:03:04.7377678Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7378393Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.264s)
2023-01-11T21:03:04.7378400Z 
2023-01-11T21:03:04.7378594Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7378670Z Ran 1 test in 0.264s
2023-01-11T21:03:04.7378676Z 
2023-01-11T21:03:04.7378747Z OK (skipped=1)
2023-01-11T21:03:04.7378752Z 
2023-01-11T21:03:04.7378838Z Generating XML reports...
2023-01-11T21:03:04.7379180Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205706.xml
2023-01-11T21:03:04.7379448Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7379739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7379864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7380144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7380278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7380283Z 
2023-01-11T21:03:04.7380356Z Running tests...
2023-01-11T21:03:04.7380552Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7380893Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s)
2023-01-11T21:03:04.7380901Z 
2023-01-11T21:03:04.7381095Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7381169Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7381174Z 
2023-01-11T21:03:04.7381235Z OK (skipped=1)
2023-01-11T21:03:04.7381240Z 
2023-01-11T21:03:04.7381323Z Generating XML reports...
2023-01-11T21:03:04.7381756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205708.xml
2023-01-11T21:03:04.7381990Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7382266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7382391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7382670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7382809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7382815Z 
2023-01-11T21:03:04.7382874Z Running tests...
2023-01-11T21:03:04.7383072Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7383407Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s)
2023-01-11T21:03:04.7383412Z 
2023-01-11T21:03:04.7383609Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7383686Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7383691Z 
2023-01-11T21:03:04.7383763Z OK (skipped=1)
2023-01-11T21:03:04.7383768Z 
2023-01-11T21:03:04.7383853Z Generating XML reports...
2023-01-11T21:03:04.7384195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205710.xml
2023-01-11T21:03:04.7384428Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7384694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7384822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7385106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7385242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7385247Z 
2023-01-11T21:03:04.7385322Z Running tests...
2023-01-11T21:03:04.7385520Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7385760Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10455
2023-01-11T21:03:04.7385919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10456
2023-01-11T21:03:04.7386074Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10457
2023-01-11T21:03:04.7386384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7386540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7386824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7386961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7387141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7387414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7387541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7387826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7387951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7388132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7388402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7388527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7388806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7388941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7389119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7389419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7389718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7389999Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7390164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7390330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7390496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7390602Z skip: Need at least 2 CUDA devices (2.175s)
2023-01-11T21:03:04.7390608Z 
2023-01-11T21:03:04.7390805Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7390881Z Ran 1 test in 2.175s
2023-01-11T21:03:04.7390887Z 
2023-01-11T21:03:04.7390959Z OK (skipped=1)
2023-01-11T21:03:04.7390967Z 
2023-01-11T21:03:04.7391052Z Generating XML reports...
2023-01-11T21:03:04.7391383Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205711.xml
2023-01-11T21:03:04.7391616Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7391889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7392015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7392296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7392431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7392437Z 
2023-01-11T21:03:04.7392510Z Running tests...
2023-01-11T21:03:04.7392705Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7393470Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.253s)
2023-01-11T21:03:04.7393477Z 
2023-01-11T21:03:04.7393674Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7393736Z Ran 1 test in 0.253s
2023-01-11T21:03:04.7393742Z 
2023-01-11T21:03:04.7393813Z OK (skipped=1)
2023-01-11T21:03:04.7393819Z 
2023-01-11T21:03:04.7393901Z Generating XML reports...
2023-01-11T21:03:04.7394242Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205715.xml
2023-01-11T21:03:04.7394474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7394748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7394876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7395156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7395293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7395298Z 
2023-01-11T21:03:04.7395374Z Running tests...
2023-01-11T21:03:04.7395568Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7395914Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s)
2023-01-11T21:03:04.7395920Z 
2023-01-11T21:03:04.7396109Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7396183Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7396191Z 
2023-01-11T21:03:04.7396261Z OK (skipped=1)
2023-01-11T21:03:04.7396266Z 
2023-01-11T21:03:04.7396350Z Generating XML reports...
2023-01-11T21:03:04.7396690Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205717.xml
2023-01-11T21:03:04.7396910Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7397184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7397307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7397587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7397722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7397728Z 
2023-01-11T21:03:04.7397801Z Running tests...
2023-01-11T21:03:04.7397998Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7398695Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.262s)
2023-01-11T21:03:04.7398702Z 
2023-01-11T21:03:04.7398893Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7398968Z Ran 1 test in 0.262s
2023-01-11T21:03:04.7398974Z 
2023-01-11T21:03:04.7399033Z OK (skipped=1)
2023-01-11T21:03:04.7399050Z 
2023-01-11T21:03:04.7399122Z Generating XML reports...
2023-01-11T21:03:04.7399457Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205719.xml
2023-01-11T21:03:04.7399688Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7399994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7400146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7400425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7400561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7400566Z 
2023-01-11T21:03:04.7400637Z Running tests...
2023-01-11T21:03:04.7400816Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7401058Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10538
2023-01-11T21:03:04.7401213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10539
2023-01-11T21:03:04.7401374Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10540
2023-01-11T21:03:04.7401648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7401770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7402050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7402185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7402362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7402712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7403006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7403133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7403416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7403551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7403730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7404004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7404129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7404396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7404529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7404705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7405006Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7405308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7405476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7405642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7405808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7405914Z skip: Need at least 2 CUDA devices (2.180s)
2023-01-11T21:03:04.7405919Z 
2023-01-11T21:03:04.7406107Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7406181Z Ran 1 test in 2.180s
2023-01-11T21:03:04.7406186Z 
2023-01-11T21:03:04.7406292Z OK (skipped=1)
2023-01-11T21:03:04.7406297Z 
2023-01-11T21:03:04.7406381Z Generating XML reports...
2023-01-11T21:03:04.7406756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205721.xml
2023-01-11T21:03:04.7406991Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7407269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7407393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7407676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7407800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7407805Z 
2023-01-11T21:03:04.7407882Z Running tests...
2023-01-11T21:03:04.7408082Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7408320Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10591
2023-01-11T21:03:04.7408480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10592
2023-01-11T21:03:04.7408637Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10593
2023-01-11T21:03:04.7408913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7409039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7409308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7409445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7409624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7409901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7410028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7410312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7410448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7410626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7410899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7411009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7411292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7411430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7411606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7411908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7412209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7412502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7412668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7412832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7412983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7413170Z skip: Need at least 2 CUDA devices (2.182s)
2023-01-11T21:03:04.7413178Z 
2023-01-11T21:03:04.7413405Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7413484Z Ran 1 test in 2.183s
2023-01-11T21:03:04.7413489Z 
2023-01-11T21:03:04.7413562Z OK (skipped=1)
2023-01-11T21:03:04.7413567Z 
2023-01-11T21:03:04.7413652Z Generating XML reports...
2023-01-11T21:03:04.7413996Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205725.xml
2023-01-11T21:03:04.7414233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7414510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7414621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7414907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7415044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7415050Z 
2023-01-11T21:03:04.7415127Z Running tests...
2023-01-11T21:03:04.7415326Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7415571Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10644
2023-01-11T21:03:04.7415730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10645
2023-01-11T21:03:04.7415886Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10646
2023-01-11T21:03:04.7416148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7416271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7416556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7416693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7416869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7417145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7417268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7417549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7417683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7417846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7418119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7418246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7418529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7418664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7418844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7419145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7419443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7419733Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7419916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7420108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7420273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7420378Z skip: Need at least 2 CUDA devices (2.177s)
2023-01-11T21:03:04.7420383Z 
2023-01-11T21:03:04.7420581Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7420656Z Ran 1 test in 2.177s
2023-01-11T21:03:04.7420661Z 
2023-01-11T21:03:04.7420732Z OK (skipped=1)
2023-01-11T21:03:04.7420738Z 
2023-01-11T21:03:04.7420821Z Generating XML reports...
2023-01-11T21:03:04.7421147Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205729.xml
2023-01-11T21:03:04.7421380Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7421767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7421896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7422179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7422316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7422322Z 
2023-01-11T21:03:04.7422397Z Running tests...
2023-01-11T21:03:04.7422595Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7423292Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.257s)
2023-01-11T21:03:04.7423301Z 
2023-01-11T21:03:04.7423496Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7423561Z Ran 1 test in 0.257s
2023-01-11T21:03:04.7423580Z 
2023-01-11T21:03:04.7423639Z OK (skipped=1)
2023-01-11T21:03:04.7423644Z 
2023-01-11T21:03:04.7423732Z Generating XML reports...
2023-01-11T21:03:04.7424072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205733.xml
2023-01-11T21:03:04.7424306Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7424581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7424706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7424989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7425130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7425135Z 
2023-01-11T21:03:04.7425198Z Running tests...
2023-01-11T21:03:04.7425397Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7426066Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.260s)
2023-01-11T21:03:04.7426072Z 
2023-01-11T21:03:04.7426273Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7426348Z Ran 1 test in 0.260s
2023-01-11T21:03:04.7426353Z 
2023-01-11T21:03:04.7426423Z OK (skipped=1)
2023-01-11T21:03:04.7426429Z 
2023-01-11T21:03:04.7426569Z Generating XML reports...
2023-01-11T21:03:04.7426907Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205735.xml
2023-01-11T21:03:04.7427174Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7427457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7427569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7427852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7427989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7427995Z 
2023-01-11T21:03:04.7428070Z Running tests...
2023-01-11T21:03:04.7428266Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7428987Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.260s)
2023-01-11T21:03:04.7428995Z 
2023-01-11T21:03:04.7429193Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7429270Z Ran 1 test in 0.260s
2023-01-11T21:03:04.7429275Z 
2023-01-11T21:03:04.7429350Z OK (skipped=1)
2023-01-11T21:03:04.7429355Z 
2023-01-11T21:03:04.7429439Z Generating XML reports...
2023-01-11T21:03:04.7429765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205737.xml
2023-01-11T21:03:04.7430001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7430279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7430402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7430684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7430820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7430825Z 
2023-01-11T21:03:04.7430898Z Running tests...
2023-01-11T21:03:04.7431094Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7431440Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.003s)
2023-01-11T21:03:04.7431457Z 
2023-01-11T21:03:04.7431639Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7431712Z Ran 1 test in 0.003s
2023-01-11T21:03:04.7431719Z 
2023-01-11T21:03:04.7431790Z OK (skipped=1)
2023-01-11T21:03:04.7431794Z 
2023-01-11T21:03:04.7431879Z Generating XML reports...
2023-01-11T21:03:04.7432220Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205739.xml
2023-01-11T21:03:04.7432453Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7432726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7432850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7433119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7433252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7433258Z 
2023-01-11T21:03:04.7433329Z Running tests...
2023-01-11T21:03:04.7433527Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7433823Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10737
2023-01-11T21:03:04.7433983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10738
2023-01-11T21:03:04.7434137Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10739
2023-01-11T21:03:04.7434412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7434534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7434803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7434937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7435113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7435391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7435516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7435796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7435929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7436105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7436378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7436489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7436766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7436902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7437081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7437380Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7437678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7437968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7438134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7438287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7438448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7438555Z skip: Need at least 3 CUDA devices (2.278s)
2023-01-11T21:03:04.7438560Z 
2023-01-11T21:03:04.7438761Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7438835Z Ran 1 test in 2.278s
2023-01-11T21:03:04.7438841Z 
2023-01-11T21:03:04.7438913Z OK (skipped=1)
2023-01-11T21:03:04.7438918Z 
2023-01-11T21:03:04.7439003Z Generating XML reports...
2023-01-11T21:03:04.7439343Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205741.xml
2023-01-11T21:03:04.7439576Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7439837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7439959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7440242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7440406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7440438Z 
2023-01-11T21:03:04.7440514Z Running tests...
2023-01-11T21:03:04.7440711Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7440947Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10790
2023-01-11T21:03:04.7441104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10791
2023-01-11T21:03:04.7441259Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10792
2023-01-11T21:03:04.7441520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7441644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7441931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7442068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7442246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7442519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7442640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7443000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7443122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7443303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7443576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7443703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7443989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7444125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7444304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7444606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7444906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7445184Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7445351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7445521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7445687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7445862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7446038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7446212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7446507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7446797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7447120Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7447214Z ok (2.282s)
2023-01-11T21:03:04.7447220Z 
2023-01-11T21:03:04.7447419Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7447494Z Ran 1 test in 2.282s
2023-01-11T21:03:04.7447500Z 
2023-01-11T21:03:04.7447562Z OK
2023-01-11T21:03:04.7447567Z 
2023-01-11T21:03:04.7447651Z Generating XML reports...
2023-01-11T21:03:04.7447996Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205745.xml
2023-01-11T21:03:04.7448231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7448505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7448616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7448901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7449041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7449047Z 
2023-01-11T21:03:04.7449120Z Running tests...
2023-01-11T21:03:04.7449319Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7449545Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10852
2023-01-11T21:03:04.7449703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10853
2023-01-11T21:03:04.7449858Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10854
2023-01-11T21:03:04.7450120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7450246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7450526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7450662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7450840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7451111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7451235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7451515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7451650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7451812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7452087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7452213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7452496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7452631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7452807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7453108Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7453403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7453695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7453884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7454090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7454250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7454414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7454589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7454761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7455056Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7455350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7455636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7455692Z ok (2.471s)
2023-01-11T21:03:04.7455697Z 
2023-01-11T21:03:04.7455895Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7455970Z Ran 1 test in 2.471s
2023-01-11T21:03:04.7455976Z 
2023-01-11T21:03:04.7456040Z OK
2023-01-11T21:03:04.7456044Z 
2023-01-11T21:03:04.7456130Z Generating XML reports...
2023-01-11T21:03:04.7456475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205749.xml
2023-01-11T21:03:04.7456707Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7456982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7457096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7457380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7457519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7457524Z 
2023-01-11T21:03:04.7457598Z Running tests...
2023-01-11T21:03:04.7457796Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7458494Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.261s)
2023-01-11T21:03:04.7458500Z 
2023-01-11T21:03:04.7458695Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7458771Z Ran 1 test in 0.261s
2023-01-11T21:03:04.7458776Z 
2023-01-11T21:03:04.7458848Z OK (skipped=1)
2023-01-11T21:03:04.7458854Z 
2023-01-11T21:03:04.7458938Z Generating XML reports...
2023-01-11T21:03:04.7459265Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205754.xml
2023-01-11T21:03:04.7459498Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7459771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7459895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7460179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7460313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7460369Z 
2023-01-11T21:03:04.7460441Z Running tests...
2023-01-11T21:03:04.7460639Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7461003Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s)
2023-01-11T21:03:04.7461010Z 
2023-01-11T21:03:04.7461191Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7461268Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7461273Z 
2023-01-11T21:03:04.7461344Z OK (skipped=1)
2023-01-11T21:03:04.7461349Z 
2023-01-11T21:03:04.7461433Z Generating XML reports...
2023-01-11T21:03:04.7461871Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205756.xml
2023-01-11T21:03:04.7462113Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7462390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7462515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7462799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7462921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7462927Z 
2023-01-11T21:03:04.7463001Z Running tests...
2023-01-11T21:03:04.7463197Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7463439Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10931
2023-01-11T21:03:04.7463601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10932
2023-01-11T21:03:04.7463757Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10933
2023-01-11T21:03:04.7464036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7464165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7464433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7464571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7464750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7465023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7465151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7465431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7465568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7465749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7466026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7466135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7466416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7466550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7466729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7467032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7467381Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7467707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7467879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7468046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7468196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7468266Z ok (2.173s)
2023-01-11T21:03:04.7468272Z 
2023-01-11T21:03:04.7468473Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7468549Z Ran 1 test in 2.173s
2023-01-11T21:03:04.7468555Z 
2023-01-11T21:03:04.7468617Z OK
2023-01-11T21:03:04.7468623Z 
2023-01-11T21:03:04.7468708Z Generating XML reports...
2023-01-11T21:03:04.7469057Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205757.xml
2023-01-11T21:03:04.7469295Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7469559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7469683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7469969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7470105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7470111Z 
2023-01-11T21:03:04.7470186Z Running tests...
2023-01-11T21:03:04.7470385Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7470609Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10984
2023-01-11T21:03:04.7470772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10985
2023-01-11T21:03:04.7470929Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10986
2023-01-11T21:03:04.7471191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7471318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7471601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7471736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7471914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7472187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7472316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7472599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7472734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7472896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7473170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7473298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7473579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7473714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7473893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7474258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7474561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7474852Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7475006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7475172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7475333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7475581Z STAGE:2023-01-11 20:58:03 10984:10984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7475827Z STAGE:2023-01-11 20:58:03 10985:10985 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7476066Z STAGE:2023-01-11 20:58:03 10986:10986 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7476317Z STAGE:2023-01-11 20:58:03 10985:10985 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7476560Z STAGE:2023-01-11 20:58:03 10986:10986 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7476805Z STAGE:2023-01-11 20:58:03 10985:10985 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7477060Z STAGE:2023-01-11 20:58:03 10986:10986 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7477302Z STAGE:2023-01-11 20:58:03 10984:10984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7477553Z STAGE:2023-01-11 20:58:03 10984:10984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7477954Z STAGE:2023-01-11 20:58:03 10985:10985 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:58:03 10986:10986 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7477960Z 
2023-01-11T21:03:04.7478206Z STAGE:2023-01-11 20:58:03 10986:10986 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7478459Z STAGE:2023-01-11 20:58:03 10986:10986 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7478701Z STAGE:2023-01-11 20:58:03 10986:10986 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7478938Z STAGE:2023-01-11 20:58:03 10984:10984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7479181Z STAGE:2023-01-11 20:58:03 10984:10984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7479420Z STAGE:2023-01-11 20:58:03 10984:10984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7479664Z STAGE:2023-01-11 20:58:03 10985:10985 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7479916Z STAGE:2023-01-11 20:58:03 10985:10985 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7480153Z STAGE:2023-01-11 20:58:03 10984:10984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7480394Z STAGE:2023-01-11 20:58:03 10984:10984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7480645Z STAGE:2023-01-11 20:58:03 10984:10984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7480885Z STAGE:2023-01-11 20:58:03 10985:10985 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7481129Z STAGE:2023-01-11 20:58:03 10985:10985 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7481384Z STAGE:2023-01-11 20:58:03 10985:10985 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7481672Z STAGE:2023-01-11 20:58:03 10986:10986 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7481924Z STAGE:2023-01-11 20:58:03 10986:10986 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7481992Z ok (2.285s)
2023-01-11T21:03:04.7481998Z 
2023-01-11T21:03:04.7482198Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7482273Z Ran 1 test in 2.285s
2023-01-11T21:03:04.7482279Z 
2023-01-11T21:03:04.7482341Z OK
2023-01-11T21:03:04.7482346Z 
2023-01-11T21:03:04.7482432Z Generating XML reports...
2023-01-11T21:03:04.7482861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205801.xml
2023-01-11T21:03:04.7483100Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7483365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7483492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7483779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7483916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7483923Z 
2023-01-11T21:03:04.7483996Z Running tests...
2023-01-11T21:03:04.7484197Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7484425Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11040
2023-01-11T21:03:04.7484583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11041
2023-01-11T21:03:04.7484726Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11042
2023-01-11T21:03:04.7485001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7485127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7485408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7485544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7485722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7485994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7493161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7493580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7493731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7493900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7494189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7494318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7494605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7494744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7494924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7495228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7495532Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7495970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7496129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7496297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7496463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7496534Z ok (2.178s)
2023-01-11T21:03:04.7496541Z 
2023-01-11T21:03:04.7496743Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7496820Z Ran 1 test in 2.178s
2023-01-11T21:03:04.7496826Z 
2023-01-11T21:03:04.7496887Z OK
2023-01-11T21:03:04.7496892Z 
2023-01-11T21:03:04.7496977Z Generating XML reports...
2023-01-11T21:03:04.7497311Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205806.xml
2023-01-11T21:03:04.7497551Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7497833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7497961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7498248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7498387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7498393Z 
2023-01-11T21:03:04.7498469Z Running tests...
2023-01-11T21:03:04.7498669Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7498855Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2023-01-11T21:03:04.7498864Z 
2023-01-11T21:03:04.7499048Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7499124Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7499131Z 
2023-01-11T21:03:04.7499204Z OK (skipped=1)
2023-01-11T21:03:04.7499210Z 
2023-01-11T21:03:04.7499295Z Generating XML reports...
2023-01-11T21:03:04.7499635Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205810.xml
2023-01-11T21:03:04.7499869Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7500144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7500267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7500539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7500678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7500683Z 
2023-01-11T21:03:04.7500756Z Running tests...
2023-01-11T21:03:04.7500956Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7501188Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11103
2023-01-11T21:03:04.7501347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11104
2023-01-11T21:03:04.7501504Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11105
2023-01-11T21:03:04.7501989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7502117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7502390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7502594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7502814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7503096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7503222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7503507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7503645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7503825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7504100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7504213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7504500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7504638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7504815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7505116Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7505414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7505706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7505872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7506037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7506189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7506368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7506545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7506717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7507013Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7507313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7507605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7508176Z STAGE:2023-01-11 20:58:13 11105:11105 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:58:13 11104:11104 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 20:58:13 11103:11103 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7508183Z 
2023-01-11T21:03:04.7508188Z 
2023-01-11T21:03:04.7508442Z STAGE:2023-01-11 20:58:13 11104:11104 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7508688Z STAGE:2023-01-11 20:58:13 11105:11105 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7508935Z STAGE:2023-01-11 20:58:13 11104:11104 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7509192Z STAGE:2023-01-11 20:58:13 11105:11105 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7509469Z STAGE:2023-01-11 20:58:13 11103:11103 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7509757Z STAGE:2023-01-11 20:58:13 11103:11103 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7510004Z STAGE:2023-01-11 20:58:13 11104:11104 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7510244Z STAGE:2023-01-11 20:58:13 11105:11105 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7510489Z STAGE:2023-01-11 20:58:13 11105:11105 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7510744Z STAGE:2023-01-11 20:58:13 11105:11105 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7510989Z STAGE:2023-01-11 20:58:13 11105:11105 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7511215Z STAGE:2023-01-11 20:58:13 11103:11103 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7511460Z STAGE:2023-01-11 20:58:13 11103:11103 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7511717Z STAGE:2023-01-11 20:58:13 11103:11103 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7511962Z STAGE:2023-01-11 20:58:13 11104:11104 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7512215Z STAGE:2023-01-11 20:58:13 11104:11104 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7512454Z STAGE:2023-01-11 20:58:13 11103:11103 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7512696Z STAGE:2023-01-11 20:58:13 11103:11103 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7512952Z STAGE:2023-01-11 20:58:13 11103:11103 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7513197Z STAGE:2023-01-11 20:58:13 11104:11104 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7513435Z STAGE:2023-01-11 20:58:13 11104:11104 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7513696Z STAGE:2023-01-11 20:58:13 11104:11104 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7513946Z STAGE:2023-01-11 20:58:13 11105:11105 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7514205Z STAGE:2023-01-11 20:58:13 11105:11105 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7514276Z ok (2.279s)
2023-01-11T21:03:04.7514282Z 
2023-01-11T21:03:04.7514483Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7514559Z Ran 1 test in 2.280s
2023-01-11T21:03:04.7514564Z 
2023-01-11T21:03:04.7514625Z OK
2023-01-11T21:03:04.7514631Z 
2023-01-11T21:03:04.7514715Z Generating XML reports...
2023-01-11T21:03:04.7515047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205811.xml
2023-01-11T21:03:04.7515284Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7515564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7515689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7515973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7516108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7516114Z 
2023-01-11T21:03:04.7516190Z Running tests...
2023-01-11T21:03:04.7516389Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7516605Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11168
2023-01-11T21:03:04.7516811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11169
2023-01-11T21:03:04.7516967Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11170
2023-01-11T21:03:04.7517273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7517398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7517677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7517813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7517988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7518264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7518375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7518657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7518795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7518974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7519247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7519374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7519660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7519799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7519979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7520269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7520574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7520868Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7521042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7521205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7521371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7521545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7521719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7521898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7522184Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7522485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7522808Z STAGE:2023-01-11 20:58:17 11170:11170 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7523061Z STAGE:2023-01-11 20:58:17 11169:11169 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7523315Z STAGE:2023-01-11 20:58:17 11170:11170 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7523578Z STAGE:2023-01-11 20:58:17 11170:11170 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7523867Z STAGE:2023-01-11 20:58:17 11169:11169 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7524190Z STAGE:2023-01-11 20:58:17 11169:11169 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7524435Z STAGE:2023-01-11 20:58:17 11170:11170 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7524661Z STAGE:2023-01-11 20:58:17 11169:11169 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7524911Z STAGE:2023-01-11 20:58:17 11169:11169 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7525170Z STAGE:2023-01-11 20:58:17 11169:11169 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7525417Z STAGE:2023-01-11 20:58:17 11170:11170 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7525672Z STAGE:2023-01-11 20:58:17 11170:11170 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7525979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7526052Z ok (2.277s)
2023-01-11T21:03:04.7526058Z 
2023-01-11T21:03:04.7526259Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7526322Z Ran 1 test in 2.277s
2023-01-11T21:03:04.7526343Z 
2023-01-11T21:03:04.7526390Z OK
2023-01-11T21:03:04.7526395Z 
2023-01-11T21:03:04.7526483Z Generating XML reports...
2023-01-11T21:03:04.7526830Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205815.xml
2023-01-11T21:03:04.7527065Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7527349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7527477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7527766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7527908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7527913Z 
2023-01-11T21:03:04.7527975Z Running tests...
2023-01-11T21:03:04.7528176Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7528405Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11229
2023-01-11T21:03:04.7528565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11230
2023-01-11T21:03:04.7528719Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11231
2023-01-11T21:03:04.7528995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7529121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7529403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7529546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7529713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7529987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7530110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7530391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7530525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7530702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7531011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7531167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7531437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7531575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7531752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7532052Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7532348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7532638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7532807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7532976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7533140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7533194Z ok (2.175s)
2023-01-11T21:03:04.7533200Z 
2023-01-11T21:03:04.7533397Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7533473Z Ran 1 test in 2.176s
2023-01-11T21:03:04.7533479Z 
2023-01-11T21:03:04.7533540Z OK
2023-01-11T21:03:04.7533546Z 
2023-01-11T21:03:04.7533630Z Generating XML reports...
2023-01-11T21:03:04.7533972Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205819.xml
2023-01-11T21:03:04.7534206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7534485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7534612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7534880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7535016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7535021Z 
2023-01-11T21:03:04.7535094Z Running tests...
2023-01-11T21:03:04.7535292Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7535991Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.259s)
2023-01-11T21:03:04.7536000Z 
2023-01-11T21:03:04.7536198Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7536274Z Ran 1 test in 0.259s
2023-01-11T21:03:04.7536279Z 
2023-01-11T21:03:04.7536352Z OK (skipped=1)
2023-01-11T21:03:04.7536357Z 
2023-01-11T21:03:04.7536441Z Generating XML reports...
2023-01-11T21:03:04.7536770Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205824.xml
2023-01-11T21:03:04.7537001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7537286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7537412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7537698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7537866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7537901Z 
2023-01-11T21:03:04.7537963Z Running tests...
2023-01-11T21:03:04.7538159Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7538387Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11292
2023-01-11T21:03:04.7538546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11293
2023-01-11T21:03:04.7538702Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11294
2023-01-11T21:03:04.7538975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7539099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7539386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7539509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7539688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7539961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7540086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7540368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7540503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7540685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7540957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7541083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7541354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7541488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7541778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7542080Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7542379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7542674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7542845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7543027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7543192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7543342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7543517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7543691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7543988Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7544281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7544634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7544706Z ok (2.376s)
2023-01-11T21:03:04.7544746Z 
2023-01-11T21:03:04.7544951Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7545028Z Ran 1 test in 2.376s
2023-01-11T21:03:04.7545034Z 
2023-01-11T21:03:04.7545082Z OK
2023-01-11T21:03:04.7545087Z 
2023-01-11T21:03:04.7545172Z Generating XML reports...
2023-01-11T21:03:04.7545516Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205826.xml
2023-01-11T21:03:04.7545752Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7546030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7546156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7546441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7546580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7546586Z 
2023-01-11T21:03:04.7546658Z Running tests...
2023-01-11T21:03:04.7546842Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7547068Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11351
2023-01-11T21:03:04.7547225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11352
2023-01-11T21:03:04.7547380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11353
2023-01-11T21:03:04.7547655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7547781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7548061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7548198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7548361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7548634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7548757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7549038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7549174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7549353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7549630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7549756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7550036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7550157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7550331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7550628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7550927Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7551218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7551417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7551615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7551780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7551835Z ok (2.287s)
2023-01-11T21:03:04.7551853Z 
2023-01-11T21:03:04.7552036Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7552112Z Ran 1 test in 2.288s
2023-01-11T21:03:04.7552117Z 
2023-01-11T21:03:04.7552177Z OK
2023-01-11T21:03:04.7552182Z 
2023-01-11T21:03:04.7552267Z Generating XML reports...
2023-01-11T21:03:04.7552608Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205830.xml
2023-01-11T21:03:04.7552842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7553120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7553246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7553515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7553650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7553656Z 
2023-01-11T21:03:04.7553727Z Running tests...
2023-01-11T21:03:04.7553925Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7554148Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11404
2023-01-11T21:03:04.7554306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11405
2023-01-11T21:03:04.7554463Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11406
2023-01-11T21:03:04.7554738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7554862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7555129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7555263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7555439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7555711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7555835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7556115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7556253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7556431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7556690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7556813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7557091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7557224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7557401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7557700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7558033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7558352Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7558519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7558671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7558834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7558900Z ok (2.380s)
2023-01-11T21:03:04.7558906Z 
2023-01-11T21:03:04.7559108Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7559182Z Ran 1 test in 2.381s
2023-01-11T21:03:04.7559188Z 
2023-01-11T21:03:04.7559249Z OK
2023-01-11T21:03:04.7559255Z 
2023-01-11T21:03:04.7559341Z Generating XML reports...
2023-01-11T21:03:04.7559680Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205834.xml
2023-01-11T21:03:04.7559918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7560181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7560304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7560585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7560719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7560724Z 
2023-01-11T21:03:04.7560797Z Running tests...
2023-01-11T21:03:04.7560996Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7561233Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11457
2023-01-11T21:03:04.7561395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11458
2023-01-11T21:03:04.7561537Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11459
2023-01-11T21:03:04.7561812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7561936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7562217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7562352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7562530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7562894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7563027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7563316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7563441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7563617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7563890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7564014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7564293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7564429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7564655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7564983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7565284Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7565566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7565733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7565896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7566061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7566236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7566413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7566590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7566889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7567183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7567460Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7567530Z ok (2.275s)
2023-01-11T21:03:04.7567535Z 
2023-01-11T21:03:04.7567734Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7567809Z Ran 1 test in 2.275s
2023-01-11T21:03:04.7567817Z 
2023-01-11T21:03:04.7567879Z OK
2023-01-11T21:03:04.7567884Z 
2023-01-11T21:03:04.7567969Z Generating XML reports...
2023-01-11T21:03:04.7568313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205838.xml
2023-01-11T21:03:04.7568546Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7568808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7568932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7569214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7569349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7569354Z 
2023-01-11T21:03:04.7569427Z Running tests...
2023-01-11T21:03:04.7569623Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7569863Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11519
2023-01-11T21:03:04.7570019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11520
2023-01-11T21:03:04.7570173Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11521
2023-01-11T21:03:04.7570436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7570559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7570843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7570979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7571157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7571460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7571615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7571899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7572022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7572196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7572468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7572591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7572871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7573009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7573190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7573491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7573792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7574071Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7574235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7574401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7574561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7574739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7574914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7575088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7575384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7575681Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7575962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7576028Z ok (2.275s)
2023-01-11T21:03:04.7576034Z 
2023-01-11T21:03:04.7576230Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7576308Z Ran 1 test in 2.275s
2023-01-11T21:03:04.7576314Z 
2023-01-11T21:03:04.7576374Z OK
2023-01-11T21:03:04.7576379Z 
2023-01-11T21:03:04.7576466Z Generating XML reports...
2023-01-11T21:03:04.7576810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205842.xml
2023-01-11T21:03:04.7577041Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7577313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7577425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7577703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7577838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7577877Z 
2023-01-11T21:03:04.7577951Z Running tests...
2023-01-11T21:03:04.7578149Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7578498Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s)
2023-01-11T21:03:04.7578505Z 
2023-01-11T21:03:04.7578700Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7578776Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7578781Z 
2023-01-11T21:03:04.7578852Z OK (skipped=1)
2023-01-11T21:03:04.7578858Z 
2023-01-11T21:03:04.7578929Z Generating XML reports...
2023-01-11T21:03:04.7579271Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205846.xml
2023-01-11T21:03:04.7579503Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7579779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7579907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7580189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7580326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7580331Z 
2023-01-11T21:03:04.7580406Z Running tests...
2023-01-11T21:03:04.7580589Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7580809Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11588
2023-01-11T21:03:04.7580966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11589
2023-01-11T21:03:04.7581121Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11590
2023-01-11T21:03:04.7581396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7581522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7581912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7582050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7582228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7582491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7582615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7582897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7583032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7583212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7583488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7583614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7583898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7584033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7584196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7584500Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7584796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7585144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7585348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7585518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7585684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7585752Z ok (2.175s)
2023-01-11T21:03:04.7585758Z 
2023-01-11T21:03:04.7585947Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7586023Z Ran 1 test in 2.175s
2023-01-11T21:03:04.7586029Z 
2023-01-11T21:03:04.7586090Z OK
2023-01-11T21:03:04.7586096Z 
2023-01-11T21:03:04.7586180Z Generating XML reports...
2023-01-11T21:03:04.7586523Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205848.xml
2023-01-11T21:03:04.7586761Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7587042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7587167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7587449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7587572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7587578Z 
2023-01-11T21:03:04.7587653Z Running tests...
2023-01-11T21:03:04.7587848Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7588067Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11641
2023-01-11T21:03:04.7588228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11642
2023-01-11T21:03:04.7588384Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11643
2023-01-11T21:03:04.7588664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7588787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7589056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7589193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7589369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7589643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7589766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7590048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7590187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7590363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7590634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7590744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7591022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7591155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7591330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7591662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7591989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7592284Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7592455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7592619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7592769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7592836Z ok (2.278s)
2023-01-11T21:03:04.7592842Z 
2023-01-11T21:03:04.7593039Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7593115Z Ran 1 test in 2.278s
2023-01-11T21:03:04.7593122Z 
2023-01-11T21:03:04.7593184Z OK
2023-01-11T21:03:04.7593189Z 
2023-01-11T21:03:04.7593274Z Generating XML reports...
2023-01-11T21:03:04.7593616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205852.xml
2023-01-11T21:03:04.7593848Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7594110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7594235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7594515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7594650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7594655Z 
2023-01-11T21:03:04.7594728Z Running tests...
2023-01-11T21:03:04.7594927Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7595170Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11694
2023-01-11T21:03:04.7595329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11695
2023-01-11T21:03:04.7595484Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11696
2023-01-11T21:03:04.7595745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7595868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7596148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7596282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7596459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7596761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7597036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7597159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7597443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7597565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7597742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7598011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7598134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7598451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7598615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7598797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7599096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7599396Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7599549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7599713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7599877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7600130Z STAGE:2023-01-11 20:58:58 11694:11694 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7600373Z STAGE:2023-01-11 20:58:58 11695:11695 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7600610Z STAGE:2023-01-11 20:58:58 11696:11696 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7600859Z STAGE:2023-01-11 20:58:58 11694:11694 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7601123Z STAGE:2023-01-11 20:58:58 11694:11694 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7601358Z STAGE:2023-01-11 20:58:58 11695:11695 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7601614Z STAGE:2023-01-11 20:58:58 11695:11695 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7601861Z STAGE:2023-01-11 20:58:58 11696:11696 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7602121Z STAGE:2023-01-11 20:58:58 11696:11696 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7602193Z ok (2.278s)
2023-01-11T21:03:04.7602198Z 
2023-01-11T21:03:04.7602395Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7602471Z Ran 1 test in 2.278s
2023-01-11T21:03:04.7602476Z 
2023-01-11T21:03:04.7602537Z OK
2023-01-11T21:03:04.7602542Z 
2023-01-11T21:03:04.7602628Z Generating XML reports...
2023-01-11T21:03:04.7603032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205856.xml
2023-01-11T21:03:04.7603270Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7603546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7603674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7603958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7604094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7604099Z 
2023-01-11T21:03:04.7604174Z Running tests...
2023-01-11T21:03:04.7604372Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7604607Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11750
2023-01-11T21:03:04.7604751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11751
2023-01-11T21:03:04.7604908Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11752
2023-01-11T21:03:04.7605183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7605346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7605657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7605795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7605970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7606245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7606356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7606636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7606771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7606952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7607227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7607350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7607628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7607764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7607939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7608224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7608526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7608821Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7608991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7609156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7609322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7609570Z STAGE:2023-01-11 20:59:02 11751:11751 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7609815Z STAGE:2023-01-11 20:59:02 11750:11750 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7610051Z STAGE:2023-01-11 20:59:02 11752:11752 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7610287Z STAGE:2023-01-11 20:59:02 11750:11750 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7610549Z STAGE:2023-01-11 20:59:02 11750:11750 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7610798Z STAGE:2023-01-11 20:59:02 11751:11751 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7611050Z STAGE:2023-01-11 20:59:02 11751:11751 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7611293Z STAGE:2023-01-11 20:59:02 11752:11752 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7611547Z STAGE:2023-01-11 20:59:02 11752:11752 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7611616Z ok (2.273s)
2023-01-11T21:03:04.7611621Z 
2023-01-11T21:03:04.7611821Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7611883Z Ran 1 test in 2.273s
2023-01-11T21:03:04.7611901Z 
2023-01-11T21:03:04.7611951Z OK
2023-01-11T21:03:04.7611956Z 
2023-01-11T21:03:04.7612041Z Generating XML reports...
2023-01-11T21:03:04.7612429Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205900.xml
2023-01-11T21:03:04.7612695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7612973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7613096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7613378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7613515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7613521Z 
2023-01-11T21:03:04.7613581Z Running tests...
2023-01-11T21:03:04.7613780Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7614123Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s)
2023-01-11T21:03:04.7614131Z 
2023-01-11T21:03:04.7614331Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7614408Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7614413Z 
2023-01-11T21:03:04.7614486Z OK (skipped=1)
2023-01-11T21:03:04.7614492Z 
2023-01-11T21:03:04.7614580Z Generating XML reports...
2023-01-11T21:03:04.7614922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205904.xml
2023-01-11T21:03:04.7615155Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7615418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7615543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7615826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7615964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7615970Z 
2023-01-11T21:03:04.7616045Z Running tests...
2023-01-11T21:03:04.7616242Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7616605Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.001s)
2023-01-11T21:03:04.7616611Z 
2023-01-11T21:03:04.7616809Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7616884Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7616889Z 
2023-01-11T21:03:04.7616948Z OK (skipped=1)
2023-01-11T21:03:04.7616953Z 
2023-01-11T21:03:04.7617038Z Generating XML reports...
2023-01-11T21:03:04.7617378Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205906.xml
2023-01-11T21:03:04.7617613Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7617891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7618016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7618296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7618435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7618441Z 
2023-01-11T21:03:04.7618512Z Running tests...
2023-01-11T21:03:04.7618694Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7618942Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11826
2023-01-11T21:03:04.7619134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11827
2023-01-11T21:03:04.7619291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11828
2023-01-11T21:03:04.7619595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7619721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7620005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7620139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7620303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7620578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7620702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7620987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7621127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7621305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7621577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7621840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7622126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7622248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7622424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7622725Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7623029Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7623325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7623493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7623660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7623823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7624086Z [E ProcessGroupGloo.cpp:138] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2023-01-11T21:03:04.7624241Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 2000 ms
2023-01-11T21:03:04.7624309Z ok (4.180s)
2023-01-11T21:03:04.7624317Z 
2023-01-11T21:03:04.7624518Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7624600Z Ran 1 test in 4.180s
2023-01-11T21:03:04.7624605Z 
2023-01-11T21:03:04.7624666Z OK
2023-01-11T21:03:04.7624671Z 
2023-01-11T21:03:04.7624755Z Generating XML reports...
2023-01-11T21:03:04.7625098Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205908.xml
2023-01-11T21:03:04.7625328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7625603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7625713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7626046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7626181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7626219Z 
2023-01-11T21:03:04.7626294Z Running tests...
2023-01-11T21:03:04.7626495Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7626732Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11879
2023-01-11T21:03:04.7626894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11880
2023-01-11T21:03:04.7627049Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11881
2023-01-11T21:03:04.7627314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7627437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7627723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7627863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7628042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7628314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7628438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7628717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7628852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7629016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7629316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7629589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7629714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7629994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7630131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7630309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7630607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7630904Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7631059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7631226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7631387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7631552Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2023-01-11T21:03:04.7631814Z [E ProcessGroupGloo.cpp:138] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2023-01-11T21:03:04.7631882Z ok (4.280s)
2023-01-11T21:03:04.7631888Z 
2023-01-11T21:03:04.7632084Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7632159Z Ran 1 test in 4.281s
2023-01-11T21:03:04.7632164Z 
2023-01-11T21:03:04.7632225Z OK
2023-01-11T21:03:04.7632298Z 
2023-01-11T21:03:04.7632371Z Generating XML reports...
2023-01-11T21:03:04.7632720Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205914.xml
2023-01-11T21:03:04.7632984Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7633263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7633389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7633673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7633810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7633815Z 
2023-01-11T21:03:04.7633889Z Running tests...
2023-01-11T21:03:04.7634072Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7634326Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11932
2023-01-11T21:03:04.7634485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11933
2023-01-11T21:03:04.7634640Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11934
2023-01-11T21:03:04.7634911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7635037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7635318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7635455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7635630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7635894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7636018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7636302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7636443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7636621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7636894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7637017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7637299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7637420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7637599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7637900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7638201Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7638498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7638666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7638829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7638991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7639206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7639395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7639569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7639866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7640161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7640456Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7640612Z [E ProcessGroupGloo.cpp:138] Rank 0 timed out in monitoredBarrier after 0 ms.
2023-01-11T21:03:04.7640736Z No ranks successfully processed in monitoredBarrier.
2023-01-11T21:03:04.7640900Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2023-01-11T21:03:04.7640967Z ok (2.376s)
2023-01-11T21:03:04.7640973Z 
2023-01-11T21:03:04.7641161Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7641236Z Ran 1 test in 2.376s
2023-01-11T21:03:04.7641242Z 
2023-01-11T21:03:04.7641302Z OK
2023-01-11T21:03:04.7641308Z 
2023-01-11T21:03:04.7641392Z Generating XML reports...
2023-01-11T21:03:04.7641739Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205920.xml
2023-01-11T21:03:04.7641973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7642249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7642376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7642718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7642850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7642858Z 
2023-01-11T21:03:04.7642933Z Running tests...
2023-01-11T21:03:04.7643133Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7643382Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11994
2023-01-11T21:03:04.7643541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11995
2023-01-11T21:03:04.7643699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11996
2023-01-11T21:03:04.7643972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7644099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7644370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7644508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7644684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7644959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7645084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7645368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7645503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7645679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7645989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7646099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7646411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7646548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7646729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7647029Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7647330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7647624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7647791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7647958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7648108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7648283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7648458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7648630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7648925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7649218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7649511Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7649680Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:03:04.7650055Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:475: UserWarning: Running monitored_barrier on global rank 2 which does not belong to the given group.
2023-01-11T21:03:04.7650175Z   f"Running {op_name} on global rank {global_rank} which does not "
2023-01-11T21:03:04.7650242Z ok (2.475s)
2023-01-11T21:03:04.7650248Z 
2023-01-11T21:03:04.7650444Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7650520Z Ran 1 test in 2.475s
2023-01-11T21:03:04.7650526Z 
2023-01-11T21:03:04.7650587Z OK
2023-01-11T21:03:04.7650592Z 
2023-01-11T21:03:04.7650676Z Generating XML reports...
2023-01-11T21:03:04.7651021Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205924.xml
2023-01-11T21:03:04.7651254Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7651532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7651644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7651924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7652061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7652067Z 
2023-01-11T21:03:04.7652140Z Running tests...
2023-01-11T21:03:04.7652338Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7652583Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12053
2023-01-11T21:03:04.7652772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12054
2023-01-11T21:03:04.7652955Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12055
2023-01-11T21:03:04.7653218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7653342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7653620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7653757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7653932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7654231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7654509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7654634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7654916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7655038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7655213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7655484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7655609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7655888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7656024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7656201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7656502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7656800Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7656953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7657115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7657279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7657446Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:03:04.7657611Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:03:04.7657777Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1, 2 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:03:04.7657845Z ok (2.375s)
2023-01-11T21:03:04.7657851Z 
2023-01-11T21:03:04.7658049Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7658112Z Ran 1 test in 2.376s
2023-01-11T21:03:04.7658129Z 
2023-01-11T21:03:04.7658178Z OK
2023-01-11T21:03:04.7658182Z 
2023-01-11T21:03:04.7658268Z Generating XML reports...
2023-01-11T21:03:04.7658611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205928.xml
2023-01-11T21:03:04.7658843Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7659115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7659288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7659600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7659736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7659742Z 
2023-01-11T21:03:04.7659803Z Running tests...
2023-01-11T21:03:04.7660001Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7660303Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T21:03:04.7660309Z 
2023-01-11T21:03:04.7660500Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7660576Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7660581Z 
2023-01-11T21:03:04.7660653Z OK (skipped=1)
2023-01-11T21:03:04.7660659Z 
2023-01-11T21:03:04.7660742Z Generating XML reports...
2023-01-11T21:03:04.7661085Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205933.xml
2023-01-11T21:03:04.7661318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7661583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7661830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7662116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7662253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7662260Z 
2023-01-11T21:03:04.7662333Z Running tests...
2023-01-11T21:03:04.7662528Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7662833Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T21:03:04.7662842Z 
2023-01-11T21:03:04.7663039Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7663102Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7663121Z 
2023-01-11T21:03:04.7663180Z OK (skipped=1)
2023-01-11T21:03:04.7663185Z 
2023-01-11T21:03:04.7663272Z Generating XML reports...
2023-01-11T21:03:04.7663618Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205934.xml
2023-01-11T21:03:04.7663860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7664136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7664260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7664546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7664685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7664690Z 
2023-01-11T21:03:04.7664752Z Running tests...
2023-01-11T21:03:04.7664948Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7665250Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T21:03:04.7665256Z 
2023-01-11T21:03:04.7665451Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7665526Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7665532Z 
2023-01-11T21:03:04.7665603Z OK (skipped=1)
2023-01-11T21:03:04.7665608Z 
2023-01-11T21:03:04.7665692Z Generating XML reports...
2023-01-11T21:03:04.7666029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205936.xml
2023-01-11T21:03:04.7666318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7666615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7666741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7667026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7667160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7667166Z 
2023-01-11T21:03:04.7667238Z Running tests...
2023-01-11T21:03:04.7667435Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7667733Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T21:03:04.7667738Z 
2023-01-11T21:03:04.7667935Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7668012Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7668018Z 
2023-01-11T21:03:04.7668074Z OK (skipped=1)
2023-01-11T21:03:04.7668079Z 
2023-01-11T21:03:04.7668166Z Generating XML reports...
2023-01-11T21:03:04.7668506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205938.xml
2023-01-11T21:03:04.7668738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7669011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7669134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7669417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7669550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7669557Z 
2023-01-11T21:03:04.7669629Z Running tests...
2023-01-11T21:03:04.7669813Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7670032Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2023-01-11T21:03:04.7670038Z 
2023-01-11T21:03:04.7670233Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7670308Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7670314Z 
2023-01-11T21:03:04.7670385Z OK (skipped=1)
2023-01-11T21:03:04.7670390Z 
2023-01-11T21:03:04.7670473Z Generating XML reports...
2023-01-11T21:03:04.7670814Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205939.xml
2023-01-11T21:03:04.7671045Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7671304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7671429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7671715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7671850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7671855Z 
2023-01-11T21:03:04.7671928Z Running tests...
2023-01-11T21:03:04.7672123Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7672304Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:03:04.7672310Z 
2023-01-11T21:03:04.7672502Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7672577Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7672582Z 
2023-01-11T21:03:04.7672640Z OK (skipped=1)
2023-01-11T21:03:04.7672645Z 
2023-01-11T21:03:04.7672766Z Generating XML reports...
2023-01-11T21:03:04.7673104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205941.xml
2023-01-11T21:03:04.7673368Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7673642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7673767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7674044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7674177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7674183Z 
2023-01-11T21:03:04.7674255Z Running tests...
2023-01-11T21:03:04.7674436Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7674631Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:03:04.7674639Z 
2023-01-11T21:03:04.7674833Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7674910Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7674915Z 
2023-01-11T21:03:04.7674987Z OK (skipped=1)
2023-01-11T21:03:04.7674992Z 
2023-01-11T21:03:04.7675075Z Generating XML reports...
2023-01-11T21:03:04.7675409Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205943.xml
2023-01-11T21:03:04.7675639Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7675897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7676018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7676295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7676432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7676438Z 
2023-01-11T21:03:04.7676511Z Running tests...
2023-01-11T21:03:04.7676704Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7676934Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:03:04.7676940Z 
2023-01-11T21:03:04.7677134Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7677207Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7677213Z 
2023-01-11T21:03:04.7677272Z OK (skipped=1)
2023-01-11T21:03:04.7677290Z 
2023-01-11T21:03:04.7677361Z Generating XML reports...
2023-01-11T21:03:04.7677694Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205944.xml
2023-01-11T21:03:04.7677927Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7678202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7678324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7678601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7678735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7678740Z 
2023-01-11T21:03:04.7678813Z Running tests...
2023-01-11T21:03:04.7678995Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7679256Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12186
2023-01-11T21:03:04.7679412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12187
2023-01-11T21:03:04.7679598Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12188
2023-01-11T21:03:04.7679898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7680022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7680302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7680436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7680612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7680871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7680993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7681276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7681413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7681593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7681866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7681989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7682269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7682389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7682564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7682972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7683276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7683572Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7683737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7683902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7684069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7684164Z skip: CUDA is not available. (2.181s)
2023-01-11T21:03:04.7684170Z 
2023-01-11T21:03:04.7684355Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7684433Z Ran 1 test in 2.182s
2023-01-11T21:03:04.7684437Z 
2023-01-11T21:03:04.7684509Z OK (skipped=1)
2023-01-11T21:03:04.7684514Z 
2023-01-11T21:03:04.7684598Z Generating XML reports...
2023-01-11T21:03:04.7684945Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205946.xml
2023-01-11T21:03:04.7685177Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7685454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7685578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7685859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7685980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7685985Z 
2023-01-11T21:03:04.7686058Z Running tests...
2023-01-11T21:03:04.7686299Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7686587Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12239
2023-01-11T21:03:04.7686746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12240
2023-01-11T21:03:04.7686900Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12241
2023-01-11T21:03:04.7687173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7687295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7687578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7687700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7687880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7688155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7688277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7688557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7688691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7688869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7689140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7689250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7689531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7689667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7689845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7690146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7690442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7690736Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7690900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7691064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7691216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7691312Z skip: CUDA is not available. (2.175s)
2023-01-11T21:03:04.7691320Z 
2023-01-11T21:03:04.7691521Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7691599Z Ran 1 test in 2.175s
2023-01-11T21:03:04.7691606Z 
2023-01-11T21:03:04.7691676Z OK (skipped=1)
2023-01-11T21:03:04.7691681Z 
2023-01-11T21:03:04.7691765Z Generating XML reports...
2023-01-11T21:03:04.7692102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205950.xml
2023-01-11T21:03:04.7692335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7692610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7692722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7693038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7693201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7693207Z 
2023-01-11T21:03:04.7693280Z Running tests...
2023-01-11T21:03:04.7693478Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7693677Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:03:04.7693683Z 
2023-01-11T21:03:04.7693879Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7693953Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7693959Z 
2023-01-11T21:03:04.7694016Z OK (skipped=1)
2023-01-11T21:03:04.7694033Z 
2023-01-11T21:03:04.7694104Z Generating XML reports...
2023-01-11T21:03:04.7694445Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205954.xml
2023-01-11T21:03:04.7694678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7694953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7695076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7695358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7695492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7695498Z 
2023-01-11T21:03:04.7695572Z Running tests...
2023-01-11T21:03:04.7695757Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7695975Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:03:04.7695984Z 
2023-01-11T21:03:04.7696174Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7696251Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7696256Z 
2023-01-11T21:03:04.7696330Z OK (skipped=1)
2023-01-11T21:03:04.7696335Z 
2023-01-11T21:03:04.7696420Z Generating XML reports...
2023-01-11T21:03:04.7696758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205956.xml
2023-01-11T21:03:04.7696990Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7697262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7697371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7697652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7697787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7697794Z 
2023-01-11T21:03:04.7697867Z Running tests...
2023-01-11T21:03:04.7698063Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7698779Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.268s)
2023-01-11T21:03:04.7698786Z 
2023-01-11T21:03:04.7698979Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7699053Z Ran 1 test in 0.268s
2023-01-11T21:03:04.7699058Z 
2023-01-11T21:03:04.7699129Z OK (skipped=1)
2023-01-11T21:03:04.7699134Z 
2023-01-11T21:03:04.7699217Z Generating XML reports...
2023-01-11T21:03:04.7699578Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205958.xml
2023-01-11T21:03:04.7699840Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7700118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7700243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7700524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7700657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7700663Z 
2023-01-11T21:03:04.7700736Z Running tests...
2023-01-11T21:03:04.7700935Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7701183Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12322
2023-01-11T21:03:04.7701331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12323
2023-01-11T21:03:04.7701488Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12324
2023-01-11T21:03:04.7701868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7701991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7702275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7702410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7702594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7702990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7703106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7703390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7703524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7703699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7703969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7704091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7704370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7704505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7704681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7704970Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7705267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7705562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7705725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7705892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7706051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7706156Z skip: Need at least 2 CUDA devices (2.278s)
2023-01-11T21:03:04.7706163Z 
2023-01-11T21:03:04.7706448Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7706524Z Ran 1 test in 2.279s
2023-01-11T21:03:04.7706530Z 
2023-01-11T21:03:04.7706588Z OK (skipped=1)
2023-01-11T21:03:04.7706629Z 
2023-01-11T21:03:04.7706715Z Generating XML reports...
2023-01-11T21:03:04.7707064Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210000.xml
2023-01-11T21:03:04.7707300Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7707576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7707700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7707983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7708119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7708128Z 
2023-01-11T21:03:04.7708189Z Running tests...
2023-01-11T21:03:04.7708385Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7708628Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12375
2023-01-11T21:03:04.7708787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12376
2023-01-11T21:03:04.7708940Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12377
2023-01-11T21:03:04.7709214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7709337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7709620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7709756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7709918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7710193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7710315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7710595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7710728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7710904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7711206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7711476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7711600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7711869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7712002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7712178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7712476Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7712770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7712937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7713135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7713293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7713413Z skip: Need at least 2 CUDA devices (2.185s)
2023-01-11T21:03:04.7713432Z 
2023-01-11T21:03:04.7713618Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7713693Z Ran 1 test in 2.186s
2023-01-11T21:03:04.7713699Z 
2023-01-11T21:03:04.7713772Z OK (skipped=1)
2023-01-11T21:03:04.7713778Z 
2023-01-11T21:03:04.7713862Z Generating XML reports...
2023-01-11T21:03:04.7714206Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210004.xml
2023-01-11T21:03:04.7714437Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7714709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7714835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7715104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7715239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7715246Z 
2023-01-11T21:03:04.7715319Z Running tests...
2023-01-11T21:03:04.7715517Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7715770Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12428
2023-01-11T21:03:04.7715926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12429
2023-01-11T21:03:04.7716079Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12430
2023-01-11T21:03:04.7716353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7716477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7716746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7716879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7717054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7717328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7717450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7717729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7717863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7718042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7718304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7718425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7718704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7718839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7719012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7719311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7719608Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7719942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7720132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7720285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7720448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7720552Z skip: Need at least 2 CUDA devices (2.172s)
2023-01-11T21:03:04.7720557Z 
2023-01-11T21:03:04.7720756Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7720831Z Ran 1 test in 2.172s
2023-01-11T21:03:04.7720837Z 
2023-01-11T21:03:04.7720907Z OK (skipped=1)
2023-01-11T21:03:04.7720912Z 
2023-01-11T21:03:04.7720994Z Generating XML reports...
2023-01-11T21:03:04.7721336Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210008.xml
2023-01-11T21:03:04.7721568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7721833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7721959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7722239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7722373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7722378Z 
2023-01-11T21:03:04.7722451Z Running tests...
2023-01-11T21:03:04.7722723Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7723429Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s)
2023-01-11T21:03:04.7723438Z 
2023-01-11T21:03:04.7723638Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7723711Z Ran 1 test in 0.254s
2023-01-11T21:03:04.7723717Z 
2023-01-11T21:03:04.7723787Z OK (skipped=1)
2023-01-11T21:03:04.7723792Z 
2023-01-11T21:03:04.7723865Z Generating XML reports...
2023-01-11T21:03:04.7724203Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210012.xml
2023-01-11T21:03:04.7724437Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7724711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7724837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7725119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7725255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7725260Z 
2023-01-11T21:03:04.7725336Z Running tests...
2023-01-11T21:03:04.7725521Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7726237Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.260s)
2023-01-11T21:03:04.7726257Z 
2023-01-11T21:03:04.7726439Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7726555Z Ran 1 test in 0.260s
2023-01-11T21:03:04.7726560Z 
2023-01-11T21:03:04.7726630Z OK (skipped=1)
2023-01-11T21:03:04.7726634Z 
2023-01-11T21:03:04.7726751Z Generating XML reports...
2023-01-11T21:03:04.7727093Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210014.xml
2023-01-11T21:03:04.7727324Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7727600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7727722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7727990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7728124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7728131Z 
2023-01-11T21:03:04.7728203Z Running tests...
2023-01-11T21:03:04.7728399Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7728668Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12501
2023-01-11T21:03:04.7728825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12502
2023-01-11T21:03:04.7728981Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12503
2023-01-11T21:03:04.7729253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7729375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7729644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7729782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7729959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7730232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7730356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7730638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7730772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7730948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7731217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7731326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7731608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7731743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7731917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7732217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7732517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7732809Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7732972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7733134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7733322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7733487Z skip: Need at least 4 CUDA devices (2.177s)
2023-01-11T21:03:04.7733495Z 
2023-01-11T21:03:04.7733696Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7733772Z Ran 1 test in 2.177s
2023-01-11T21:03:04.7733778Z 
2023-01-11T21:03:04.7733849Z OK (skipped=1)
2023-01-11T21:03:04.7733855Z 
2023-01-11T21:03:04.7733940Z Generating XML reports...
2023-01-11T21:03:04.7734280Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210016.xml
2023-01-11T21:03:04.7734513Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7734775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7734902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7735184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7735320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7735325Z 
2023-01-11T21:03:04.7735398Z Running tests...
2023-01-11T21:03:04.7735596Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7735879Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12554
2023-01-11T21:03:04.7736042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12555
2023-01-11T21:03:04.7736197Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12556
2023-01-11T21:03:04.7736455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7736581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7736867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7737006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7737184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7737461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7737584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7737864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7737998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7738166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7738436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7738559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7738837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7738971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7739149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7739448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7739745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7740074Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7740278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7740443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7740608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7740714Z skip: Need at least 4 CUDA devices (2.171s)
2023-01-11T21:03:04.7740719Z 
2023-01-11T21:03:04.7740920Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7740994Z Ran 1 test in 2.171s
2023-01-11T21:03:04.7741000Z 
2023-01-11T21:03:04.7741072Z OK (skipped=1)
2023-01-11T21:03:04.7741077Z 
2023-01-11T21:03:04.7741160Z Generating XML reports...
2023-01-11T21:03:04.7741486Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210020.xml
2023-01-11T21:03:04.7741830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7742111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7742236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7742518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7742655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7742661Z 
2023-01-11T21:03:04.7742734Z Running tests...
2023-01-11T21:03:04.7742933Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7743646Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.257s)
2023-01-11T21:03:04.7743655Z 
2023-01-11T21:03:04.7743852Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7743914Z Ran 1 test in 0.257s
2023-01-11T21:03:04.7743932Z 
2023-01-11T21:03:04.7743991Z OK (skipped=1)
2023-01-11T21:03:04.7743996Z 
2023-01-11T21:03:04.7744079Z Generating XML reports...
2023-01-11T21:03:04.7744422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210024.xml
2023-01-11T21:03:04.7744655Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7744931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7745058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7745341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7745479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7745485Z 
2023-01-11T21:03:04.7745544Z Running tests...
2023-01-11T21:03:04.7745743Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7745979Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12617
2023-01-11T21:03:04.7746137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12618
2023-01-11T21:03:04.7746293Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12619
2023-01-11T21:03:04.7746562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7746742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7747060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7747182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7747358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7747663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7747935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7748059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7748338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7748474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7748652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7748925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7749035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7749314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7749447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7749622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7749920Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7750217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7750382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7750548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7750710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7750872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7751045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7751217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7751516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7751816Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7752108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7752358Z STAGE:2023-01-11 21:00:28 12619:12619 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7752597Z STAGE:2023-01-11 21:00:28 12617:12617 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7752832Z STAGE:2023-01-11 21:00:28 12618:12618 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7753067Z STAGE:2023-01-11 21:00:28 12619:12619 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7753310Z STAGE:2023-01-11 21:00:28 12618:12618 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7753603Z STAGE:2023-01-11 21:00:28 12617:12617 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7753887Z STAGE:2023-01-11 21:00:28 12619:12619 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7754145Z STAGE:2023-01-11 21:00:28 12618:12618 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7754398Z STAGE:2023-01-11 21:00:28 12617:12617 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7754639Z STAGE:2023-01-11 21:00:28 12618:12618 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7754876Z STAGE:2023-01-11 21:00:28 12619:12619 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7755111Z STAGE:2023-01-11 21:00:28 12617:12617 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7755342Z STAGE:2023-01-11 21:00:28 12619:12619 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7755600Z STAGE:2023-01-11 21:00:28 12619:12619 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7755848Z STAGE:2023-01-11 21:00:28 12617:12617 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7756097Z STAGE:2023-01-11 21:00:28 12618:12618 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7756351Z STAGE:2023-01-11 21:00:28 12617:12617 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7756605Z STAGE:2023-01-11 21:00:28 12618:12618 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7756847Z STAGE:2023-01-11 21:00:28 12619:12619 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7757087Z STAGE:2023-01-11 21:00:28 12617:12617 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7757325Z STAGE:2023-01-11 21:00:28 12618:12618 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7757560Z STAGE:2023-01-11 21:00:28 12617:12617 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7757804Z STAGE:2023-01-11 21:00:28 12618:12618 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7758044Z STAGE:2023-01-11 21:00:28 12619:12619 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7758300Z STAGE:2023-01-11 21:00:28 12617:12617 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7758553Z STAGE:2023-01-11 21:00:28 12618:12618 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7758808Z STAGE:2023-01-11 21:00:28 12619:12619 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7758876Z ok (2.276s)
2023-01-11T21:03:04.7758882Z 
2023-01-11T21:03:04.7759080Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7759158Z Ran 1 test in 2.276s
2023-01-11T21:03:04.7759164Z 
2023-01-11T21:03:04.7759211Z OK
2023-01-11T21:03:04.7759216Z 
2023-01-11T21:03:04.7759299Z Generating XML reports...
2023-01-11T21:03:04.7759646Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210026.xml
2023-01-11T21:03:04.7759880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7760155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7760278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7760559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7760693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7760698Z 
2023-01-11T21:03:04.7760771Z Running tests...
2023-01-11T21:03:04.7760990Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7761251Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12682
2023-01-11T21:03:04.7761411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12683
2023-01-11T21:03:04.7761565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12684
2023-01-11T21:03:04.7761840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7761962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7762239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7762372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7762538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7762890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7763016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7763296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7763431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7763608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7763908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7764181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7764305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7764571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7764707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7764879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7765179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7765474Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7765639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7765803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7765966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7766140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7766302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7766477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7766775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7767073Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7767320Z STAGE:2023-01-11 21:00:32 12683:12683 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7767561Z STAGE:2023-01-11 21:00:32 12684:12684 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7767844Z STAGE:2023-01-11 21:00:32 12684:12684 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7768128Z STAGE:2023-01-11 21:00:32 12684:12684 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7768373Z STAGE:2023-01-11 21:00:32 12684:12684 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7768656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7768899Z STAGE:2023-01-11 21:00:32 12682:12682 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7769313Z STAGE:2023-01-11 21:00:32 12682:12682 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:00:32 12683:12683 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7769319Z 
2023-01-11T21:03:04.7769576Z STAGE:2023-01-11 21:00:32 12682:12682 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7769833Z STAGE:2023-01-11 21:00:32 12683:12683 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7770076Z STAGE:2023-01-11 21:00:32 12683:12683 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7770314Z STAGE:2023-01-11 21:00:32 12682:12682 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7770561Z STAGE:2023-01-11 21:00:32 12684:12684 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7770983Z STAGE:2023-01-11 21:00:32 12682:12682 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:00:32 12684:12684 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7770988Z 
2023-01-11T21:03:04.7771241Z STAGE:2023-01-11 21:00:32 12682:12682 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7771471Z STAGE:2023-01-11 21:00:32 12683:12683 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7771731Z STAGE:2023-01-11 21:00:32 12683:12683 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7771971Z STAGE:2023-01-11 21:00:32 12682:12682 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7772209Z STAGE:2023-01-11 21:00:32 12684:12684 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7772451Z STAGE:2023-01-11 21:00:32 12683:12683 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7772697Z STAGE:2023-01-11 21:00:32 12682:12682 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7772937Z STAGE:2023-01-11 21:00:32 12683:12683 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7773193Z STAGE:2023-01-11 21:00:32 12682:12682 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7773440Z STAGE:2023-01-11 21:00:32 12684:12684 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7773683Z STAGE:2023-01-11 21:00:32 12683:12683 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7773933Z STAGE:2023-01-11 21:00:32 12684:12684 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7774004Z ok (2.278s)
2023-01-11T21:03:04.7774009Z 
2023-01-11T21:03:04.7774208Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7774285Z Ran 1 test in 2.278s
2023-01-11T21:03:04.7774290Z 
2023-01-11T21:03:04.7774352Z OK
2023-01-11T21:03:04.7774357Z 
2023-01-11T21:03:04.7774442Z Generating XML reports...
2023-01-11T21:03:04.7774785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210030.xml
2023-01-11T21:03:04.7775018Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7775319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7775470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7775753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7775890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7775896Z 
2023-01-11T21:03:04.7775968Z Running tests...
2023-01-11T21:03:04.7776166Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7776406Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12747
2023-01-11T21:03:04.7776564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12748
2023-01-11T21:03:04.7776705Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12749
2023-01-11T21:03:04.7776981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7777106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7777386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7777520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7777697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7777971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7778093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7778373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7778496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7778674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7778948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7779072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7779352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7779486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7779661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7779962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7780261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7780547Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7780712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7780876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7781038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7781209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7781385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7781556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7782008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7782352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7782634Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7782882Z STAGE:2023-01-11 21:00:36 12747:12747 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7783290Z STAGE:2023-01-11 21:00:36 12748:12748 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:00:36 12749:12749 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7783297Z 
2023-01-11T21:03:04.7783542Z STAGE:2023-01-11 21:00:36 12749:12749 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7783794Z STAGE:2023-01-11 21:00:36 12748:12748 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7784039Z STAGE:2023-01-11 21:00:36 12747:12747 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7784297Z STAGE:2023-01-11 21:00:36 12749:12749 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7784555Z STAGE:2023-01-11 21:00:36 12748:12748 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7784809Z STAGE:2023-01-11 21:00:36 12747:12747 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7785051Z STAGE:2023-01-11 21:00:36 12749:12749 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7785276Z STAGE:2023-01-11 21:00:36 12748:12748 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7785512Z STAGE:2023-01-11 21:00:36 12747:12747 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7785761Z STAGE:2023-01-11 21:00:36 12749:12749 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7786019Z STAGE:2023-01-11 21:00:36 12749:12749 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7786267Z STAGE:2023-01-11 21:00:36 12747:12747 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7786511Z STAGE:2023-01-11 21:00:36 12748:12748 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7786768Z STAGE:2023-01-11 21:00:36 12747:12747 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7787022Z STAGE:2023-01-11 21:00:36 12748:12748 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7787265Z STAGE:2023-01-11 21:00:36 12749:12749 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7787489Z STAGE:2023-01-11 21:00:36 12747:12747 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7787730Z STAGE:2023-01-11 21:00:36 12748:12748 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7787979Z STAGE:2023-01-11 21:00:36 12747:12747 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7788233Z STAGE:2023-01-11 21:00:36 12747:12747 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7788478Z STAGE:2023-01-11 21:00:36 12748:12748 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7788723Z STAGE:2023-01-11 21:00:36 12749:12749 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7788978Z STAGE:2023-01-11 21:00:36 12748:12748 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7789232Z STAGE:2023-01-11 21:00:36 12749:12749 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7789300Z ok (2.274s)
2023-01-11T21:03:04.7789348Z 
2023-01-11T21:03:04.7789536Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7789613Z Ran 1 test in 2.274s
2023-01-11T21:03:04.7789619Z 
2023-01-11T21:03:04.7789711Z OK
2023-01-11T21:03:04.7789717Z 
2023-01-11T21:03:04.7789803Z Generating XML reports...
2023-01-11T21:03:04.7790148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210034.xml
2023-01-11T21:03:04.7790381Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7790654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7790778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7791047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7791184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7791192Z 
2023-01-11T21:03:04.7791265Z Running tests...
2023-01-11T21:03:04.7791468Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7791705Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12812
2023-01-11T21:03:04.7791863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12813
2023-01-11T21:03:04.7792019Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12814
2023-01-11T21:03:04.7792292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7792414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7792681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7792819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7792996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7793269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7793391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7793671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7793805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7793983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7794238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7794371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7794815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7794954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7795130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7795433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7795733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7796026Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7796192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7796385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7796549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7796757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7796931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7797106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7797401Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7797694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7797946Z STAGE:2023-01-11 21:00:40 12814:12814 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7798194Z STAGE:2023-01-11 21:00:40 12813:12813 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7798432Z STAGE:2023-01-11 21:00:40 12814:12814 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7798691Z STAGE:2023-01-11 21:00:40 12814:12814 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7798938Z STAGE:2023-01-11 21:00:40 12814:12814 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7799239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7799485Z STAGE:2023-01-11 21:00:40 12812:12812 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7799734Z STAGE:2023-01-11 21:00:40 12812:12812 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7799979Z STAGE:2023-01-11 21:00:40 12813:12813 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7800238Z STAGE:2023-01-11 21:00:40 12812:12812 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7800495Z STAGE:2023-01-11 21:00:40 12813:12813 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7800723Z STAGE:2023-01-11 21:00:40 12813:12813 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7800960Z STAGE:2023-01-11 21:00:40 12812:12812 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7801208Z STAGE:2023-01-11 21:00:40 12814:12814 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7801463Z STAGE:2023-01-11 21:00:40 12814:12814 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7801707Z STAGE:2023-01-11 21:00:40 12812:12812 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7801960Z STAGE:2023-01-11 21:00:40 12812:12812 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7802209Z STAGE:2023-01-11 21:00:40 12813:12813 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7802461Z STAGE:2023-01-11 21:00:40 12813:12813 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7802774Z STAGE:2023-01-11 21:00:40 12812:12812 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7803001Z STAGE:2023-01-11 21:00:40 12814:12814 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7803240Z STAGE:2023-01-11 21:00:40 12813:12813 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7803656Z STAGE:2023-01-11 21:00:40 12813:12813 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:00:40 12812:12812 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7803663Z 
2023-01-11T21:03:04.7803968Z STAGE:2023-01-11 21:00:40 12812:12812 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7804253Z STAGE:2023-01-11 21:00:40 12813:12813 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7804501Z STAGE:2023-01-11 21:00:40 12814:12814 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7804758Z STAGE:2023-01-11 21:00:40 12814:12814 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7804825Z ok (2.278s)
2023-01-11T21:03:04.7804830Z 
2023-01-11T21:03:04.7805031Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7805105Z Ran 1 test in 2.279s
2023-01-11T21:03:04.7805110Z 
2023-01-11T21:03:04.7805156Z OK
2023-01-11T21:03:04.7805161Z 
2023-01-11T21:03:04.7805244Z Generating XML reports...
2023-01-11T21:03:04.7805591Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210038.xml
2023-01-11T21:03:04.7805827Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7806108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7806234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7806517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7806655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7806661Z 
2023-01-11T21:03:04.7806734Z Running tests...
2023-01-11T21:03:04.7806920Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7807151Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12877
2023-01-11T21:03:04.7807309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12878
2023-01-11T21:03:04.7807466Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12879
2023-01-11T21:03:04.7807743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7807867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7808149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7808283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7808447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7808721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7808844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7809126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7809262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7809438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7809711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7809833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7810112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7810233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7810406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7810742Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7811070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7811369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7811535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7811711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7811872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7812029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7812186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7812360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7812657Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7812952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7813199Z STAGE:2023-01-11 21:00:44 12879:12879 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7813441Z STAGE:2023-01-11 21:00:44 12878:12878 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7813689Z STAGE:2023-01-11 21:00:44 12878:12878 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7813931Z STAGE:2023-01-11 21:00:44 12879:12879 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7814187Z STAGE:2023-01-11 21:00:44 12878:12878 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7814437Z STAGE:2023-01-11 21:00:44 12879:12879 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7814681Z STAGE:2023-01-11 21:00:44 12879:12879 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7814978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7815221Z STAGE:2023-01-11 21:00:44 12878:12878 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7815467Z STAGE:2023-01-11 21:00:44 12878:12878 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7815710Z STAGE:2023-01-11 21:00:44 12879:12879 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7815966Z STAGE:2023-01-11 21:00:44 12878:12878 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7816228Z STAGE:2023-01-11 21:00:44 12879:12879 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7816295Z ok (2.376s)
2023-01-11T21:03:04.7816301Z 
2023-01-11T21:03:04.7816491Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7816569Z Ran 1 test in 2.377s
2023-01-11T21:03:04.7816574Z 
2023-01-11T21:03:04.7816639Z OK
2023-01-11T21:03:04.7816644Z 
2023-01-11T21:03:04.7816730Z Generating XML reports...
2023-01-11T21:03:04.7817072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210042.xml
2023-01-11T21:03:04.7817307Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7817578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7817701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7818000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7818163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7818169Z 
2023-01-11T21:03:04.7818242Z Running tests...
2023-01-11T21:03:04.7818442Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7818671Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12938
2023-01-11T21:03:04.7818830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12939
2023-01-11T21:03:04.7818984Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12940
2023-01-11T21:03:04.7819258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7819378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7819644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7819780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7819957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7820226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7820349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7820626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7820760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7820933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7821194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7821317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7821598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7821843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7822023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7822324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7822624Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7822918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7823087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7823240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7823403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7823576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7823753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7823926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7824221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7824514Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7824828Z STAGE:2023-01-11 21:00:48 12939:12939 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7825121Z STAGE:2023-01-11 21:00:48 12940:12940 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7825361Z STAGE:2023-01-11 21:00:48 12940:12940 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7825606Z STAGE:2023-01-11 21:00:48 12939:12939 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7825866Z STAGE:2023-01-11 21:00:48 12940:12940 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7826123Z STAGE:2023-01-11 21:00:48 12939:12939 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7826368Z STAGE:2023-01-11 21:00:48 12940:12940 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7826611Z STAGE:2023-01-11 21:00:48 12939:12939 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7826861Z STAGE:2023-01-11 21:00:48 12939:12939 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7827105Z STAGE:2023-01-11 21:00:48 12940:12940 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7827360Z STAGE:2023-01-11 21:00:48 12939:12939 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7827599Z STAGE:2023-01-11 21:00:48 12940:12940 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7827902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7827970Z ok (2.278s)
2023-01-11T21:03:04.7827976Z 
2023-01-11T21:03:04.7828173Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7828247Z Ran 1 test in 2.278s
2023-01-11T21:03:04.7828256Z 
2023-01-11T21:03:04.7828317Z OK
2023-01-11T21:03:04.7828322Z 
2023-01-11T21:03:04.7828405Z Generating XML reports...
2023-01-11T21:03:04.7828751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210047.xml
2023-01-11T21:03:04.7828982Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7829244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7829368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7829647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7829781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7829786Z 
2023-01-11T21:03:04.7829858Z Running tests...
2023-01-11T21:03:04.7830055Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7830295Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12999
2023-01-11T21:03:04.7830453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13000
2023-01-11T21:03:04.7830608Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13001
2023-01-11T21:03:04.7830868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7830989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7831266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7831399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7831576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7831912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7832216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7832340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7832603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7832738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7832911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7833183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7833306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7833590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7833726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7833901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7834197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7834480Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7834642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7834806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7834968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7835141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7835315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7835487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7835784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7836081Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7836476Z STAGE:2023-01-11 21:00:53 13001:13001 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:00:53 13000:13000 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7836496Z 
2023-01-11T21:03:04.7836893Z STAGE:2023-01-11 21:00:53 13000:13000 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:00:53 13001:13001 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7836917Z 
2023-01-11T21:03:04.7837334Z STAGE:2023-01-11 21:00:53 13001:13001 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:00:53 13000:13000 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7837351Z 
2023-01-11T21:03:04.7837582Z STAGE:2023-01-11 21:00:53 13001:13001 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7837821Z STAGE:2023-01-11 21:00:53 13000:13000 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7838069Z STAGE:2023-01-11 21:00:53 13000:13000 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7838310Z STAGE:2023-01-11 21:00:53 13001:13001 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7838604Z STAGE:2023-01-11 21:00:53 13000:13000 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7838893Z STAGE:2023-01-11 21:00:53 13001:13001 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7839196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7839263Z ok (2.277s)
2023-01-11T21:03:04.7839269Z 
2023-01-11T21:03:04.7839453Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7839527Z Ran 1 test in 2.277s
2023-01-11T21:03:04.7839533Z 
2023-01-11T21:03:04.7839592Z OK
2023-01-11T21:03:04.7839597Z 
2023-01-11T21:03:04.7839683Z Generating XML reports...
2023-01-11T21:03:04.7840029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210051.xml
2023-01-11T21:03:04.7840264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7840544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7840672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7840952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7841075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7841081Z 
2023-01-11T21:03:04.7841154Z Running tests...
2023-01-11T21:03:04.7841353Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7841586Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13060
2023-01-11T21:03:04.7841745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13061
2023-01-11T21:03:04.7841900Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13062
2023-01-11T21:03:04.7842178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7842305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7842574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7842774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7842955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7843258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7843531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7843653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7843935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7844071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7844246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7844503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7844628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7844908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7845043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7845219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7845588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7845913Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7846079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7846246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7846394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7846569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7846743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7846915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7847215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7847514Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7847764Z STAGE:2023-01-11 21:00:57 13061:13061 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7848005Z STAGE:2023-01-11 21:00:57 13062:13062 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7848254Z STAGE:2023-01-11 21:00:57 13062:13062 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7848485Z STAGE:2023-01-11 21:00:57 13061:13061 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7848744Z STAGE:2023-01-11 21:00:57 13062:13062 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7849007Z STAGE:2023-01-11 21:00:57 13061:13061 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7849252Z STAGE:2023-01-11 21:00:57 13062:13062 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7849491Z STAGE:2023-01-11 21:00:57 13061:13061 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7849902Z STAGE:2023-01-11 21:00:57 13062:13062 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:00:57 13061:13061 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7849908Z 
2023-01-11T21:03:04.7850335Z STAGE:2023-01-11 21:00:57 13061:13061 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:00:57 13062:13062 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7850341Z 
2023-01-11T21:03:04.7850640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7850713Z ok (2.276s)
2023-01-11T21:03:04.7850719Z 
2023-01-11T21:03:04.7850919Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7850983Z Ran 1 test in 2.276s
2023-01-11T21:03:04.7851004Z 
2023-01-11T21:03:04.7851052Z OK
2023-01-11T21:03:04.7851057Z 
2023-01-11T21:03:04.7851140Z Generating XML reports...
2023-01-11T21:03:04.7851484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210055.xml
2023-01-11T21:03:04.7851717Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7851993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7852117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7852396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7852572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7852578Z 
2023-01-11T21:03:04.7852677Z Running tests...
2023-01-11T21:03:04.7852876Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7853101Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13121
2023-01-11T21:03:04.7853258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13122
2023-01-11T21:03:04.7853410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13123
2023-01-11T21:03:04.7853686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7853810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7854090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7854228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7854394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7854672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7854795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7855079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7855215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7855393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7855668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7855796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7856062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7856196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7856371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7856672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7856972Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7857263Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7857428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7857682Z STAGE:2023-01-11 21:01:01 13121:13121 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7857847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7857997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7858243Z STAGE:2023-01-11 21:01:01 13122:13122 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7858486Z STAGE:2023-01-11 21:01:01 13123:13123 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7858735Z STAGE:2023-01-11 21:01:01 13123:13123 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7858979Z STAGE:2023-01-11 21:01:01 13122:13122 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7859236Z STAGE:2023-01-11 21:01:01 13123:13123 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7859529Z STAGE:2023-01-11 21:01:01 13122:13122 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7859802Z STAGE:2023-01-11 21:01:01 13121:13121 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7860060Z STAGE:2023-01-11 21:01:01 13121:13121 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7860288Z STAGE:2023-01-11 21:01:01 13123:13123 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7860526Z STAGE:2023-01-11 21:01:01 13122:13122 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7860761Z STAGE:2023-01-11 21:01:01 13121:13121 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7861007Z STAGE:2023-01-11 21:01:01 13123:13123 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7861262Z STAGE:2023-01-11 21:01:01 13123:13123 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7861512Z STAGE:2023-01-11 21:01:01 13121:13121 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7861848Z STAGE:2023-01-11 21:01:01 13122:13122 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7862106Z STAGE:2023-01-11 21:01:01 13121:13121 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7862369Z STAGE:2023-01-11 21:01:01 13122:13122 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7862595Z STAGE:2023-01-11 21:01:01 13123:13123 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7862832Z STAGE:2023-01-11 21:01:01 13121:13121 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7863067Z STAGE:2023-01-11 21:01:01 13122:13122 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7863317Z STAGE:2023-01-11 21:01:01 13123:13123 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7863573Z STAGE:2023-01-11 21:01:01 13123:13123 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7863992Z STAGE:2023-01-11 21:01:01 13122:13122 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:01:01 13121:13121 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7863998Z 
2023-01-11T21:03:04.7864431Z STAGE:2023-01-11 21:01:01 13121:13121 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:01:01 13122:13122 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7864438Z 
2023-01-11T21:03:04.7864506Z ok (2.377s)
2023-01-11T21:03:04.7864512Z 
2023-01-11T21:03:04.7864711Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7864786Z Ran 1 test in 2.378s
2023-01-11T21:03:04.7864793Z 
2023-01-11T21:03:04.7864840Z OK
2023-01-11T21:03:04.7864859Z 
2023-01-11T21:03:04.7864930Z Generating XML reports...
2023-01-11T21:03:04.7865274Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210059.xml
2023-01-11T21:03:04.7865508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7865786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7865912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7866196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7866332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7866338Z 
2023-01-11T21:03:04.7866409Z Running tests...
2023-01-11T21:03:04.7866592Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7866875Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13177
2023-01-11T21:03:04.7867064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13178
2023-01-11T21:03:04.7867221Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13179
2023-01-11T21:03:04.7867497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7867622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7867904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7868040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7868218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7868478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7868604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7868886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7869021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7869244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7869514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7869637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7869918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7870040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7870217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7870518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7870816Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7871109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7871273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7871436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7871597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7871847Z STAGE:2023-01-11 21:01:05 13177:13177 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7872078Z STAGE:2023-01-11 21:01:05 13179:13179 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7872320Z STAGE:2023-01-11 21:01:05 13178:13178 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7872568Z STAGE:2023-01-11 21:01:05 13179:13179 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7872812Z STAGE:2023-01-11 21:01:05 13178:13178 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7873069Z STAGE:2023-01-11 21:01:05 13179:13179 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7873324Z STAGE:2023-01-11 21:01:05 13178:13178 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7873571Z STAGE:2023-01-11 21:01:05 13177:13177 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7873860Z STAGE:2023-01-11 21:01:05 13177:13177 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7874287Z STAGE:2023-01-11 21:01:05 13179:13179 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:01:05 13178:13178 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7874293Z 
2023-01-11T21:03:04.7874535Z STAGE:2023-01-11 21:01:05 13177:13177 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7874768Z STAGE:2023-01-11 21:01:05 13179:13179 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7875008Z STAGE:2023-01-11 21:01:05 13177:13177 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7875261Z STAGE:2023-01-11 21:01:05 13179:13179 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7875515Z STAGE:2023-01-11 21:01:05 13177:13177 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7875798Z STAGE:2023-01-11 21:01:05 13178:13178 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7876054Z STAGE:2023-01-11 21:01:05 13178:13178 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7876296Z STAGE:2023-01-11 21:01:05 13179:13179 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7876531Z STAGE:2023-01-11 21:01:05 13177:13177 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7876766Z STAGE:2023-01-11 21:01:05 13178:13178 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7876998Z STAGE:2023-01-11 21:01:05 13177:13177 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7877239Z STAGE:2023-01-11 21:01:05 13178:13178 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7877494Z STAGE:2023-01-11 21:01:05 13177:13177 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7877741Z STAGE:2023-01-11 21:01:05 13179:13179 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7877998Z STAGE:2023-01-11 21:01:05 13178:13178 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7878248Z STAGE:2023-01-11 21:01:05 13179:13179 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7878318Z ok (2.175s)
2023-01-11T21:03:04.7878324Z 
2023-01-11T21:03:04.7878524Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7878600Z Ran 1 test in 2.175s
2023-01-11T21:03:04.7878605Z 
2023-01-11T21:03:04.7878653Z OK
2023-01-11T21:03:04.7878658Z 
2023-01-11T21:03:04.7878741Z Generating XML reports...
2023-01-11T21:03:04.7879081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210103.xml
2023-01-11T21:03:04.7879317Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7879594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7879718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7879998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7880133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7880140Z 
2023-01-11T21:03:04.7880200Z Running tests...
2023-01-11T21:03:04.7880398Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7880614Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2023-01-11T21:03:04.7880619Z 
2023-01-11T21:03:04.7880801Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7880907Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7880913Z 
2023-01-11T21:03:04.7880984Z OK (skipped=1)
2023-01-11T21:03:04.7880989Z 
2023-01-11T21:03:04.7881099Z Generating XML reports...
2023-01-11T21:03:04.7881441Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210107.xml
2023-01-11T21:03:04.7881670Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7881943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7882069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7882346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7882467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7882475Z 
2023-01-11T21:03:04.7882551Z Running tests...
2023-01-11T21:03:04.7882808Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7883042Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13243
2023-01-11T21:03:04.7883201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13244
2023-01-11T21:03:04.7883357Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13245
2023-01-11T21:03:04.7883636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7883760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7884024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7884162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7884342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7884648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7884921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7885043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7885326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7885460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7885636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7885896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7886021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7886303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7886437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7886616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7886917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7887217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7887382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7887547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7887735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7888016Z STAGE:2023-01-11 21:01:11 13243:13243 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7888260Z STAGE:2023-01-11 21:01:11 13245:13245 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7888500Z STAGE:2023-01-11 21:01:11 13244:13244 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7888749Z STAGE:2023-01-11 21:01:11 13243:13243 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7888995Z STAGE:2023-01-11 21:01:11 13245:13245 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7889238Z STAGE:2023-01-11 21:01:11 13244:13244 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7889498Z STAGE:2023-01-11 21:01:11 13243:13243 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7889937Z STAGE:2023-01-11 21:01:11 13244:13244 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:01:11 13245:13245 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7889944Z 
2023-01-11T21:03:04.7890338Z STAGE:2023-01-11 21:01:11 13244:13244 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:01:11 13245:13245 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7890344Z 
2023-01-11T21:03:04.7890570Z STAGE:2023-01-11 21:01:11 13243:13243 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7890817Z STAGE:2023-01-11 21:01:11 13245:13245 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7891073Z STAGE:2023-01-11 21:01:11 13245:13245 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7891322Z STAGE:2023-01-11 21:01:11 13243:13243 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7891580Z STAGE:2023-01-11 21:01:11 13243:13243 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7891825Z STAGE:2023-01-11 21:01:11 13244:13244 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7892082Z STAGE:2023-01-11 21:01:11 13244:13244 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7892321Z STAGE:2023-01-11 21:01:11 13243:13243 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7892559Z STAGE:2023-01-11 21:01:11 13245:13245 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7892780Z STAGE:2023-01-11 21:01:11 13244:13244 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7893025Z STAGE:2023-01-11 21:01:11 13243:13243 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7893281Z STAGE:2023-01-11 21:01:11 13243:13243 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7893528Z STAGE:2023-01-11 21:01:11 13244:13244 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7893772Z STAGE:2023-01-11 21:01:11 13245:13245 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7894203Z STAGE:2023-01-11 21:01:11 13245:13245 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:01:11 13244:13244 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7894209Z 
2023-01-11T21:03:04.7894274Z ok (2.274s)
2023-01-11T21:03:04.7894280Z 
2023-01-11T21:03:04.7894479Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7894553Z Ran 1 test in 2.274s
2023-01-11T21:03:04.7894559Z 
2023-01-11T21:03:04.7894619Z OK
2023-01-11T21:03:04.7894624Z 
2023-01-11T21:03:04.7894696Z Generating XML reports...
2023-01-11T21:03:04.7895076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210109.xml
2023-01-11T21:03:04.7895342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7895620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7895743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7896024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7896158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7896164Z 
2023-01-11T21:03:04.7896236Z Running tests...
2023-01-11T21:03:04.7896421Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7896638Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2023-01-11T21:03:04.7896645Z 
2023-01-11T21:03:04.7896839Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7896916Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7896920Z 
2023-01-11T21:03:04.7896993Z OK (skipped=1)
2023-01-11T21:03:04.7896998Z 
2023-01-11T21:03:04.7897081Z Generating XML reports...
2023-01-11T21:03:04.7897421Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210113.xml
2023-01-11T21:03:04.7897655Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7897927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7898037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7898321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7898458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7898463Z 
2023-01-11T21:03:04.7898540Z Running tests...
2023-01-11T21:03:04.7898737Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7898940Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.002s)
2023-01-11T21:03:04.7898946Z 
2023-01-11T21:03:04.7899139Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7899213Z Ran 1 test in 0.003s
2023-01-11T21:03:04.7899219Z 
2023-01-11T21:03:04.7899289Z OK (skipped=1)
2023-01-11T21:03:04.7899294Z 
2023-01-11T21:03:04.7899364Z Generating XML reports...
2023-01-11T21:03:04.7899698Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210115.xml
2023-01-11T21:03:04.7899926Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7900199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7900324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7900602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7900736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7900742Z 
2023-01-11T21:03:04.7900813Z Running tests...
2023-01-11T21:03:04.7901006Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7901217Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13319
2023-01-11T21:03:04.7901374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13320
2023-01-11T21:03:04.7901571Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13321
2023-01-11T21:03:04.7902012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7902139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7902420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7902557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7902735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7902995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7903116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7903401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7903540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7903721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7904024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7904301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7904424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7904704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7904825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7904999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7905298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7905596Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7905765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7905931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7906094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7906343Z STAGE:2023-01-11 21:01:18 13319:13319 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7906582Z STAGE:2023-01-11 21:01:18 13320:13320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7906812Z STAGE:2023-01-11 21:01:18 13321:13321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7907059Z STAGE:2023-01-11 21:01:18 13321:13321 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7907482Z STAGE:2023-01-11 21:01:18 13321:13321 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:01:18 13320:13320 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7907488Z 
2023-01-11T21:03:04.7907735Z STAGE:2023-01-11 21:01:18 13319:13319 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7907991Z STAGE:2023-01-11 21:01:18 13320:13320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7908247Z STAGE:2023-01-11 21:01:18 13319:13319 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7908486Z STAGE:2023-01-11 21:01:18 13321:13321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7908762Z STAGE:2023-01-11 21:01:18 13320:13320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7908995Z STAGE:2023-01-11 21:01:18 13319:13319 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7909269Z STAGE:2023-01-11 21:01:18 13321:13321 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7909515Z STAGE:2023-01-11 21:01:18 13321:13321 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7909760Z STAGE:2023-01-11 21:01:18 13319:13319 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7910001Z STAGE:2023-01-11 21:01:18 13320:13320 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7910256Z STAGE:2023-01-11 21:01:18 13319:13319 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7910507Z STAGE:2023-01-11 21:01:18 13320:13320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7910745Z STAGE:2023-01-11 21:01:18 13321:13321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7910984Z STAGE:2023-01-11 21:01:18 13319:13319 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7911216Z STAGE:2023-01-11 21:01:18 13320:13320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7911447Z STAGE:2023-01-11 21:01:18 13320:13320 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7911694Z STAGE:2023-01-11 21:01:18 13319:13319 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7911935Z STAGE:2023-01-11 21:01:18 13321:13321 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7912192Z STAGE:2023-01-11 21:01:18 13320:13320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7912446Z STAGE:2023-01-11 21:01:18 13319:13319 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7912702Z STAGE:2023-01-11 21:01:18 13321:13321 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7912771Z ok (2.286s)
2023-01-11T21:03:04.7912779Z 
2023-01-11T21:03:04.7912978Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7913052Z Ran 1 test in 2.286s
2023-01-11T21:03:04.7913058Z 
2023-01-11T21:03:04.7913106Z OK
2023-01-11T21:03:04.7913124Z 
2023-01-11T21:03:04.7913195Z Generating XML reports...
2023-01-11T21:03:04.7913535Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210116.xml
2023-01-11T21:03:04.7913768Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7914043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7914166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7914447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7914587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7914592Z 
2023-01-11T21:03:04.7914663Z Running tests...
2023-01-11T21:03:04.7914846Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7915033Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2023-01-11T21:03:04.7915039Z 
2023-01-11T21:03:04.7915233Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7915306Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7915312Z 
2023-01-11T21:03:04.7915382Z OK (skipped=1)
2023-01-11T21:03:04.7915387Z 
2023-01-11T21:03:04.7915472Z Generating XML reports...
2023-01-11T21:03:04.7915814Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210121.xml
2023-01-11T21:03:04.7916077Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7916376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7916489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7916769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7916902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7916907Z 
2023-01-11T21:03:04.7916978Z Running tests...
2023-01-11T21:03:04.7917176Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7917366Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2023-01-11T21:03:04.7917372Z 
2023-01-11T21:03:04.7917567Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7917703Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7917708Z 
2023-01-11T21:03:04.7917766Z OK (skipped=1)
2023-01-11T21:03:04.7917785Z 
2023-01-11T21:03:04.7917856Z Generating XML reports...
2023-01-11T21:03:04.7918195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210122.xml
2023-01-11T21:03:04.7918427Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7918696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7918819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7919096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7919231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7919238Z 
2023-01-11T21:03:04.7919310Z Running tests...
2023-01-11T21:03:04.7919493Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7919725Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13395
2023-01-11T21:03:04.7919882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13396
2023-01-11T21:03:04.7920036Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13397
2023-01-11T21:03:04.7920311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7920433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7920710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7920846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7921011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7921285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7921408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7921688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7921820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7922000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7922271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7922394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7922775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7922934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7923110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7923414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7923712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7924008Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7924173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7924337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7924503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7924755Z STAGE:2023-01-11 21:01:26 13395:13395 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7924983Z STAGE:2023-01-11 21:01:26 13396:13396 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7925223Z STAGE:2023-01-11 21:01:26 13397:13397 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7925470Z STAGE:2023-01-11 21:01:26 13396:13396 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7925711Z STAGE:2023-01-11 21:01:26 13397:13397 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7926132Z STAGE:2023-01-11 21:01:26 13396:13396 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:01:26 13395:13395 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7926140Z 
2023-01-11T21:03:04.7926394Z STAGE:2023-01-11 21:01:26 13397:13397 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7926654Z STAGE:2023-01-11 21:01:26 13395:13395 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7926893Z STAGE:2023-01-11 21:01:26 13396:13396 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7927130Z STAGE:2023-01-11 21:01:26 13397:13397 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7927364Z STAGE:2023-01-11 21:01:26 13395:13395 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7927596Z STAGE:2023-01-11 21:01:26 13397:13397 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7927849Z STAGE:2023-01-11 21:01:26 13397:13397 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7928259Z STAGE:2023-01-11 21:01:26 13395:13395 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:01:26 13396:13396 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7928267Z 
2023-01-11T21:03:04.7928694Z STAGE:2023-01-11 21:01:26 13396:13396 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:01:26 13395:13395 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7928700Z 
2023-01-11T21:03:04.7928939Z STAGE:2023-01-11 21:01:26 13397:13397 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7929177Z STAGE:2023-01-11 21:01:26 13395:13395 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7929411Z STAGE:2023-01-11 21:01:26 13396:13396 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7929654Z STAGE:2023-01-11 21:01:26 13396:13396 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7929931Z STAGE:2023-01-11 21:01:26 13395:13395 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7930217Z STAGE:2023-01-11 21:01:26 13396:13396 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7930459Z STAGE:2023-01-11 21:01:26 13395:13395 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7930704Z STAGE:2023-01-11 21:01:26 13397:13397 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7930956Z STAGE:2023-01-11 21:01:26 13397:13397 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7931023Z ok (2.278s)
2023-01-11T21:03:04.7931029Z 
2023-01-11T21:03:04.7931229Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7931304Z Ran 1 test in 2.278s
2023-01-11T21:03:04.7931310Z 
2023-01-11T21:03:04.7931369Z OK
2023-01-11T21:03:04.7931374Z 
2023-01-11T21:03:04.7931459Z Generating XML reports...
2023-01-11T21:03:04.7931800Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210124.xml
2023-01-11T21:03:04.7932022Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7932297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7932420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7932698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7932832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7932839Z 
2023-01-11T21:03:04.7932911Z Running tests...
2023-01-11T21:03:04.7933108Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7933332Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13451
2023-01-11T21:03:04.7933489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13452
2023-01-11T21:03:04.7933632Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13453
2023-01-11T21:03:04.7933907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7934031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7934314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7934448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7934622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7934894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7935018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7935289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7935421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7935596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7935868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7935989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7936268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7936400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7936607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7936935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7937222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7937515Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7937677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7937842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7938002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7938249Z STAGE:2023-01-11 21:01:30 13451:13451 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7938489Z STAGE:2023-01-11 21:01:30 13452:13452 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7938734Z STAGE:2023-01-11 21:01:30 13453:13453 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7938979Z STAGE:2023-01-11 21:01:30 13452:13452 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7939209Z STAGE:2023-01-11 21:01:30 13453:13453 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7939627Z STAGE:2023-01-11 21:01:30 13452:13452 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:01:30 13451:13451 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7939633Z 
2023-01-11T21:03:04.7939890Z STAGE:2023-01-11 21:01:30 13453:13453 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7940145Z STAGE:2023-01-11 21:01:30 13451:13451 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7940388Z STAGE:2023-01-11 21:01:30 13453:13453 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7940628Z STAGE:2023-01-11 21:01:30 13451:13451 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7940861Z STAGE:2023-01-11 21:01:30 13452:13452 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7941107Z STAGE:2023-01-11 21:01:30 13453:13453 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7941349Z STAGE:2023-01-11 21:01:30 13452:13452 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7941607Z STAGE:2023-01-11 21:01:30 13453:13453 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7942133Z STAGE:2023-01-11 21:01:30 13451:13451 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:01:30 13452:13452 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7942154Z 
2023-01-11T21:03:04.7942396Z STAGE:2023-01-11 21:01:30 13451:13451 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7942638Z STAGE:2023-01-11 21:01:30 13453:13453 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7942877Z STAGE:2023-01-11 21:01:30 13452:13452 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7943114Z STAGE:2023-01-11 21:01:30 13451:13451 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7943361Z STAGE:2023-01-11 21:01:30 13453:13453 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7943609Z STAGE:2023-01-11 21:01:30 13451:13451 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7944031Z STAGE:2023-01-11 21:01:30 13453:13453 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:01:30 13452:13452 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7944090Z 
2023-01-11T21:03:04.7944382Z STAGE:2023-01-11 21:01:30 13451:13451 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7944641Z STAGE:2023-01-11 21:01:30 13452:13452 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7944697Z ok (2.277s)
2023-01-11T21:03:04.7944702Z 
2023-01-11T21:03:04.7944902Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7944979Z Ran 1 test in 2.277s
2023-01-11T21:03:04.7944984Z 
2023-01-11T21:03:04.7945046Z OK
2023-01-11T21:03:04.7945051Z 
2023-01-11T21:03:04.7945136Z Generating XML reports...
2023-01-11T21:03:04.7945480Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210128.xml
2023-01-11T21:03:04.7945715Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7945993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7946121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7946391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7946527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7946533Z 
2023-01-11T21:03:04.7946607Z Running tests...
2023-01-11T21:03:04.7946803Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7947035Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13507
2023-01-11T21:03:04.7947196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13508
2023-01-11T21:03:04.7947352Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13509
2023-01-11T21:03:04.7947631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7947742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7948028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7948163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7948340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7948613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7948738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7949021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7949158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7949337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7949594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7949717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7949995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7950132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7950309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7950607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7950948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7951312Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7951477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7951629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7951793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7951863Z ok (2.181s)
2023-01-11T21:03:04.7951868Z 
2023-01-11T21:03:04.7952068Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7952145Z Ran 1 test in 2.181s
2023-01-11T21:03:04.7952151Z 
2023-01-11T21:03:04.7952213Z OK
2023-01-11T21:03:04.7952218Z 
2023-01-11T21:03:04.7952303Z Generating XML reports...
2023-01-11T21:03:04.7952647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210132.xml
2023-01-11T21:03:04.7952868Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7953144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7953268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7953552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7953688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7953693Z 
2023-01-11T21:03:04.7953767Z Running tests...
2023-01-11T21:03:04.7953963Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7954199Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13560
2023-01-11T21:03:04.7954358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13561
2023-01-11T21:03:04.7954503Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13562
2023-01-11T21:03:04.7954776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7954899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7955183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7955317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7955492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7955761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7955885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7956153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7956285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7956461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7956735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7956857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7957135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7957268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7957471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7957796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7958080Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7958372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7958536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7958696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7958858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7959105Z STAGE:2023-01-11 21:01:38 13561:13561 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7959348Z STAGE:2023-01-11 21:01:38 13562:13562 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7959586Z STAGE:2023-01-11 21:01:38 13560:13560 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7959834Z STAGE:2023-01-11 21:01:38 13560:13560 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7960064Z STAGE:2023-01-11 21:01:38 13561:13561 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7960302Z STAGE:2023-01-11 21:01:38 13562:13562 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7960559Z STAGE:2023-01-11 21:01:38 13560:13560 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7960814Z STAGE:2023-01-11 21:01:38 13561:13561 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7961068Z STAGE:2023-01-11 21:01:38 13562:13562 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7961307Z STAGE:2023-01-11 21:01:38 13562:13562 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7961544Z STAGE:2023-01-11 21:01:38 13561:13561 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7961778Z STAGE:2023-01-11 21:01:38 13560:13560 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7962187Z STAGE:2023-01-11 21:01:38 13562:13562 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:01:38 13561:13561 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7962420Z STAGE:2023-01-11 21:01:38 13560:13560 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7962437Z 
2023-01-11T21:03:04.7962742Z STAGE:2023-01-11 21:01:38 13562:13562 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7963175Z STAGE:2023-01-11 21:01:38 13561:13561 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:01:38 13560:13560 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7963185Z 
2023-01-11T21:03:04.7963429Z STAGE:2023-01-11 21:01:38 13560:13560 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7963672Z STAGE:2023-01-11 21:01:38 13562:13562 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7963913Z STAGE:2023-01-11 21:01:38 13561:13561 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7964161Z STAGE:2023-01-11 21:01:38 13560:13560 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7964417Z STAGE:2023-01-11 21:01:38 13560:13560 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7964824Z STAGE:2023-01-11 21:01:38 13561:13561 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:01:38 13562:13562 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7964862Z 
2023-01-11T21:03:04.7965145Z STAGE:2023-01-11 21:01:38 13561:13561 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7965404Z STAGE:2023-01-11 21:01:38 13562:13562 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7965459Z ok (2.276s)
2023-01-11T21:03:04.7965477Z 
2023-01-11T21:03:04.7965661Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7965734Z Ran 1 test in 2.276s
2023-01-11T21:03:04.7965741Z 
2023-01-11T21:03:04.7965800Z OK
2023-01-11T21:03:04.7965804Z 
2023-01-11T21:03:04.7965888Z Generating XML reports...
2023-01-11T21:03:04.7966229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210136.xml
2023-01-11T21:03:04.7966461Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7966740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7966864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7967132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7967266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7967272Z 
2023-01-11T21:03:04.7967344Z Running tests...
2023-01-11T21:03:04.7967543Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7967727Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2023-01-11T21:03:04.7967733Z 
2023-01-11T21:03:04.7967928Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7968003Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7968009Z 
2023-01-11T21:03:04.7968081Z OK (skipped=1)
2023-01-11T21:03:04.7968086Z 
2023-01-11T21:03:04.7968170Z Generating XML reports...
2023-01-11T21:03:04.7968496Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210140.xml
2023-01-11T21:03:04.7968727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7968999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7969122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7969401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7969534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7969539Z 
2023-01-11T21:03:04.7969610Z Running tests...
2023-01-11T21:03:04.7969806Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7969986Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2023-01-11T21:03:04.7970005Z 
2023-01-11T21:03:04.7970185Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7970257Z Ran 1 test in 0.002s
2023-01-11T21:03:04.7970262Z 
2023-01-11T21:03:04.7970332Z OK (skipped=1)
2023-01-11T21:03:04.7970337Z 
2023-01-11T21:03:04.7970420Z Generating XML reports...
2023-01-11T21:03:04.7970760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210142.xml
2023-01-11T21:03:04.7970990Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7971261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7971383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7971682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7971841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7971848Z 
2023-01-11T21:03:04.7971920Z Running tests...
2023-01-11T21:03:04.7972117Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7972348Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13636
2023-01-11T21:03:04.7972505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13637
2023-01-11T21:03:04.7972659Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13638
2023-01-11T21:03:04.7972929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7973050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7973319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7973454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7973630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7973903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7974025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7974304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7974438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7974616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7974877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7974999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7975278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7975410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7975586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7975884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7976181Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7976481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7976647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7976801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7976960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7977134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7977307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7977477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7977771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7978066Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7978416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7978667Z STAGE:2023-01-11 21:01:46 13637:13637 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7978896Z STAGE:2023-01-11 21:01:46 13638:13638 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7979129Z STAGE:2023-01-11 21:01:46 13636:13636 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7979375Z STAGE:2023-01-11 21:01:46 13636:13636 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7979786Z STAGE:2023-01-11 21:01:46 13637:13637 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:01:46 13638:13638 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7979795Z 
2023-01-11T21:03:04.7980400Z STAGE:2023-01-11 21:01:46 13636:13636 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:01:46 13637:13637 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:01:46 13638:13638 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7980407Z 
2023-01-11T21:03:04.7980411Z 
2023-01-11T21:03:04.7980653Z STAGE:2023-01-11 21:01:46 13636:13636 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7980893Z STAGE:2023-01-11 21:01:46 13638:13638 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7981129Z STAGE:2023-01-11 21:01:46 13637:13637 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7981378Z STAGE:2023-01-11 21:01:46 13637:13637 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7981619Z STAGE:2023-01-11 21:01:46 13636:13636 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7982000Z STAGE:2023-01-11 21:01:46 13637:13637 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7982424Z STAGE:2023-01-11 21:01:46 13638:13638 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:01:46 13636:13636 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7982430Z 
2023-01-11T21:03:04.7982675Z STAGE:2023-01-11 21:01:46 13638:13638 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7982914Z STAGE:2023-01-11 21:01:46 13636:13636 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7983159Z STAGE:2023-01-11 21:01:46 13637:13637 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7983395Z STAGE:2023-01-11 21:01:46 13638:13638 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7983642Z STAGE:2023-01-11 21:01:46 13638:13638 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7983886Z STAGE:2023-01-11 21:01:46 13636:13636 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7984308Z STAGE:2023-01-11 21:01:46 13637:13637 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:01:46 13638:13638 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7984314Z 
2023-01-11T21:03:04.7984569Z STAGE:2023-01-11 21:01:46 13636:13636 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7984825Z STAGE:2023-01-11 21:01:46 13637:13637 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7984893Z ok (2.275s)
2023-01-11T21:03:04.7984899Z 
2023-01-11T21:03:04.7985084Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7985157Z Ran 1 test in 2.275s
2023-01-11T21:03:04.7985163Z 
2023-01-11T21:03:04.7985227Z OK
2023-01-11T21:03:04.7985280Z 
2023-01-11T21:03:04.7985368Z Generating XML reports...
2023-01-11T21:03:04.7985748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210144.xml
2023-01-11T21:03:04.7985986Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7986263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7986388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7986670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7986793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7986798Z 
2023-01-11T21:03:04.7986872Z Running tests...
2023-01-11T21:03:04.7987075Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7987307Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13701
2023-01-11T21:03:04.7987467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13702
2023-01-11T21:03:04.7987622Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13703
2023-01-11T21:03:04.7987896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7988018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7988287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7988421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7988596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.7988873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7988997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7989278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7989411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7989586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.7989856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7989965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7990245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7990377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7990554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.7990858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7991159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7991453Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.7991616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.7991779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.7991927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.7992128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:03:04.7992329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:03:04.7992502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:03:04.7992797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7993091Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7993495Z STAGE:2023-01-11 21:01:50 13702:13702 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:01:50 13703:13703 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7993501Z 
2023-01-11T21:03:04.7993913Z STAGE:2023-01-11 21:01:50 13702:13702 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:01:50 13703:13703 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7993921Z 
2023-01-11T21:03:04.7994353Z STAGE:2023-01-11 21:01:50 13703:13703 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:01:50 13702:13702 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7994359Z 
2023-01-11T21:03:04.7994604Z STAGE:2023-01-11 21:01:50 13702:13702 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7994840Z STAGE:2023-01-11 21:01:50 13703:13703 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.7995074Z STAGE:2023-01-11 21:01:50 13703:13703 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7995315Z STAGE:2023-01-11 21:01:50 13702:13702 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.7995571Z STAGE:2023-01-11 21:01:50 13703:13703 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7995828Z STAGE:2023-01-11 21:01:50 13702:13702 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.7996129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:03:04.7996196Z ok (2.280s)
2023-01-11T21:03:04.7996201Z 
2023-01-11T21:03:04.7996400Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7996475Z Ran 1 test in 2.280s
2023-01-11T21:03:04.7996481Z 
2023-01-11T21:03:04.7996540Z OK
2023-01-11T21:03:04.7996545Z 
2023-01-11T21:03:04.7996616Z Generating XML reports...
2023-01-11T21:03:04.7996958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210148.xml
2023-01-11T21:03:04.7997189Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.7997471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7997594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7997878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7998013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7998018Z 
2023-01-11T21:03:04.7998091Z Running tests...
2023-01-11T21:03:04.7998276Z ----------------------------------------------------------------------
2023-01-11T21:03:04.7998504Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13762
2023-01-11T21:03:04.7998659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13763
2023-01-11T21:03:04.7998813Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13764
2023-01-11T21:03:04.7999128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.7999280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.7999561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.7999695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.7999873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8000132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8000254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8000535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8000671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8000849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8001120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8001243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8001525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8001659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8001822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8002123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8002423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8002799Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8002967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8003132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8003296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8003365Z ok (2.180s)
2023-01-11T21:03:04.8003371Z 
2023-01-11T21:03:04.8003571Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8003632Z Ran 1 test in 2.181s
2023-01-11T21:03:04.8003638Z 
2023-01-11T21:03:04.8003700Z OK
2023-01-11T21:03:04.8003705Z 
2023-01-11T21:03:04.8003791Z Generating XML reports...
2023-01-11T21:03:04.8004135Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210152.xml
2023-01-11T21:03:04.8004371Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8004647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8004771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8005053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8005174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8005193Z 
2023-01-11T21:03:04.8005253Z Running tests...
2023-01-11T21:03:04.8005449Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8005674Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13815
2023-01-11T21:03:04.8005869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13816
2023-01-11T21:03:04.8006054Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13817
2023-01-11T21:03:04.8006332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8006453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8006732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8006855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8007035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8007308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8007434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8007718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8007855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8008033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8008335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8008591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8008716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8008996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8009132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8009307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8009606Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8009903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8010069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8010232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8010381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8010451Z ok (2.179s)
2023-01-11T21:03:04.8010456Z 
2023-01-11T21:03:04.8010657Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8010736Z Ran 1 test in 2.179s
2023-01-11T21:03:04.8010742Z 
2023-01-11T21:03:04.8010804Z OK
2023-01-11T21:03:04.8010809Z 
2023-01-11T21:03:04.8010896Z Generating XML reports...
2023-01-11T21:03:04.8011238Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210156.xml
2023-01-11T21:03:04.8011471Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8011748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8011859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8012141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8012281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8012315Z 
2023-01-11T21:03:04.8012390Z Running tests...
2023-01-11T21:03:04.8012592Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8012853Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13868
2023-01-11T21:03:04.8013012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13869
2023-01-11T21:03:04.8013166Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13870
2023-01-11T21:03:04.8013425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8013550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8013831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8013966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8014147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8014418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8014542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8014823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8014959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8015123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8015393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8015513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8015788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8015924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8016102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8016403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8016700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8016992Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8017144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8017301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8017466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8017534Z ok (2.177s)
2023-01-11T21:03:04.8017539Z 
2023-01-11T21:03:04.8017740Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8017813Z Ran 1 test in 2.178s
2023-01-11T21:03:04.8017818Z 
2023-01-11T21:03:04.8017877Z OK
2023-01-11T21:03:04.8017883Z 
2023-01-11T21:03:04.8017965Z Generating XML reports...
2023-01-11T21:03:04.8018295Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210200.xml
2023-01-11T21:03:04.8018525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8018797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8018921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8019230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8019393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8019399Z 
2023-01-11T21:03:04.8019471Z Running tests...
2023-01-11T21:03:04.8019668Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8019925Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13921
2023-01-11T21:03:04.8020069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13922
2023-01-11T21:03:04.8020221Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13923
2023-01-11T21:03:04.8020493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8020615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8020900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8021036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8021213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8021513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8021900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8022009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8022292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8022427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8022605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8022879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8023003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8023281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8023418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8023580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8023879Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8024178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8024346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8024511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8024676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8024924Z STAGE:2023-01-11 21:02:06 13921:13921 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8025166Z STAGE:2023-01-11 21:02:06 13923:13923 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8025402Z STAGE:2023-01-11 21:02:06 13922:13922 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8033522Z STAGE:2023-01-11 21:02:06 13921:13921 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8033893Z STAGE:2023-01-11 21:02:06 13921:13921 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8034279Z STAGE:2023-01-11 21:02:06 13922:13922 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8034576Z STAGE:2023-01-11 21:02:06 13923:13923 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8034839Z STAGE:2023-01-11 21:02:06 13922:13922 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8035095Z STAGE:2023-01-11 21:02:06 13923:13923 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8035165Z ok (2.279s)
2023-01-11T21:03:04.8035173Z 
2023-01-11T21:03:04.8035377Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8035455Z Ran 1 test in 2.279s
2023-01-11T21:03:04.8035461Z 
2023-01-11T21:03:04.8035523Z OK
2023-01-11T21:03:04.8035528Z 
2023-01-11T21:03:04.8035601Z Generating XML reports...
2023-01-11T21:03:04.8035949Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210204.xml
2023-01-11T21:03:04.8036187Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8036470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8036598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8036885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8037022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8037029Z 
2023-01-11T21:03:04.8037102Z Running tests...
2023-01-11T21:03:04.8037290Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8037541Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13977
2023-01-11T21:03:04.8037706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13978
2023-01-11T21:03:04.8037867Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13979
2023-01-11T21:03:04.8038144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8038266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8038549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8038688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8038867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8039130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8039257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8039544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8039681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8039864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8040137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8040262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8040544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8040677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8040841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8041176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8041506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8041807Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8041976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8042142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8042307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8042555Z STAGE:2023-01-11 21:02:10 13977:13977 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8042884Z STAGE:2023-01-11 21:02:10 13978:13978 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8043114Z STAGE:2023-01-11 21:02:10 13979:13979 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8043363Z STAGE:2023-01-11 21:02:10 13977:13977 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8043622Z STAGE:2023-01-11 21:02:10 13977:13977 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8043870Z STAGE:2023-01-11 21:02:10 13978:13978 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8044114Z STAGE:2023-01-11 21:02:10 13979:13979 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8044373Z STAGE:2023-01-11 21:02:10 13978:13978 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8044627Z STAGE:2023-01-11 21:02:10 13979:13979 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8044698Z ok (2.271s)
2023-01-11T21:03:04.8044705Z 
2023-01-11T21:03:04.8044888Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8044968Z Ran 1 test in 2.271s
2023-01-11T21:03:04.8044974Z 
2023-01-11T21:03:04.8045035Z OK
2023-01-11T21:03:04.8045041Z 
2023-01-11T21:03:04.8045126Z Generating XML reports...
2023-01-11T21:03:04.8045473Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210208.xml
2023-01-11T21:03:04.8045708Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8045987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8046114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8046396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8046523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8046530Z 
2023-01-11T21:03:04.8046605Z Running tests...
2023-01-11T21:03:04.8046806Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8047051Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14033
2023-01-11T21:03:04.8047209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14034
2023-01-11T21:03:04.8047367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14035
2023-01-11T21:03:04.8047642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8047765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8048048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8048205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8048414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8048694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8048819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8049101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8049237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8049419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8049689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8049803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8050086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8050221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8050399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8050703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8051005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8051298Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8051464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8051633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8051788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8052037Z STAGE:2023-01-11 21:02:14 14033:14033 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8052282Z STAGE:2023-01-11 21:02:14 14034:14034 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8052521Z STAGE:2023-01-11 21:02:14 14035:14035 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8052770Z STAGE:2023-01-11 21:02:14 14035:14035 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8053030Z STAGE:2023-01-11 21:02:14 14035:14035 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8053279Z STAGE:2023-01-11 21:02:14 14033:14033 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8053537Z STAGE:2023-01-11 21:02:14 14033:14033 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8053786Z STAGE:2023-01-11 21:02:14 14034:14034 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8054027Z STAGE:2023-01-11 21:02:14 14034:14034 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8054095Z ok (2.179s)
2023-01-11T21:03:04.8054101Z 
2023-01-11T21:03:04.8054299Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8054374Z Ran 1 test in 2.179s
2023-01-11T21:03:04.8054380Z 
2023-01-11T21:03:04.8054441Z OK
2023-01-11T21:03:04.8054447Z 
2023-01-11T21:03:04.8054532Z Generating XML reports...
2023-01-11T21:03:04.8054875Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210212.xml
2023-01-11T21:03:04.8055112Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8055420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8055572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8055860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8055996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8056001Z 
2023-01-11T21:03:04.8056076Z Running tests...
2023-01-11T21:03:04.8056279Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8056454Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2023-01-11T21:03:04.8056460Z 
2023-01-11T21:03:04.8056652Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8056728Z Ran 1 test in 0.002s
2023-01-11T21:03:04.8056736Z 
2023-01-11T21:03:04.8056796Z OK (skipped=1)
2023-01-11T21:03:04.8056801Z 
2023-01-11T21:03:04.8056886Z Generating XML reports...
2023-01-11T21:03:04.8057225Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210216.xml
2023-01-11T21:03:04.8057462Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8057735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8057859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8058142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8058279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8058284Z 
2023-01-11T21:03:04.8058357Z Running tests...
2023-01-11T21:03:04.8058544Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8058739Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2023-01-11T21:03:04.8058745Z 
2023-01-11T21:03:04.8058943Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8059019Z Ran 1 test in 0.002s
2023-01-11T21:03:04.8059025Z 
2023-01-11T21:03:04.8059096Z OK (skipped=1)
2023-01-11T21:03:04.8059102Z 
2023-01-11T21:03:04.8059185Z Generating XML reports...
2023-01-11T21:03:04.8059529Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210218.xml
2023-01-11T21:03:04.8059763Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8060039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8060153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8060440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8060579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8060584Z 
2023-01-11T21:03:04.8060657Z Running tests...
2023-01-11T21:03:04.8060853Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8061043Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2023-01-11T21:03:04.8061049Z 
2023-01-11T21:03:04.8061241Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8061317Z Ran 1 test in 0.002s
2023-01-11T21:03:04.8061322Z 
2023-01-11T21:03:04.8061380Z OK (skipped=1)
2023-01-11T21:03:04.8061397Z 
2023-01-11T21:03:04.8061469Z Generating XML reports...
2023-01-11T21:03:04.8061994Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210220.xml
2023-01-11T21:03:04.8062283Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8062595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8062721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8063006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8063143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8063148Z 
2023-01-11T21:03:04.8063221Z Running tests...
2023-01-11T21:03:04.8063406Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8063649Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14119
2023-01-11T21:03:04.8063812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14120
2023-01-11T21:03:04.8063972Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14121
2023-01-11T21:03:04.8064250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8064374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8064657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8064793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8064959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8065234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8065360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8065647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8065785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8065963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8066237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8066364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8066646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8066768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8066947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8067250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8067552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8067846Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8068013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8068181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8068348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8068598Z STAGE:2023-01-11 21:02:23 14119:14119 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8068890Z STAGE:2023-01-11 21:02:23 14120:14120 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8069159Z STAGE:2023-01-11 21:02:23 14121:14121 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8069410Z STAGE:2023-01-11 21:02:23 14121:14121 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8069669Z STAGE:2023-01-11 21:02:23 14121:14121 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8069920Z STAGE:2023-01-11 21:02:23 14119:14119 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8070178Z STAGE:2023-01-11 21:02:23 14119:14119 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8070423Z STAGE:2023-01-11 21:02:23 14120:14120 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8070678Z STAGE:2023-01-11 21:02:23 14120:14120 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8070750Z ok (2.179s)
2023-01-11T21:03:04.8070755Z 
2023-01-11T21:03:04.8070939Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8071019Z Ran 1 test in 2.180s
2023-01-11T21:03:04.8071025Z 
2023-01-11T21:03:04.8071086Z OK
2023-01-11T21:03:04.8071091Z 
2023-01-11T21:03:04.8071177Z Generating XML reports...
2023-01-11T21:03:04.8071521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210221.xml
2023-01-11T21:03:04.8071756Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8072035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8072161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8072431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8072571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8072576Z 
2023-01-11T21:03:04.8072651Z Running tests...
2023-01-11T21:03:04.8072854Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8073086Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14175
2023-01-11T21:03:04.8073249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14176
2023-01-11T21:03:04.8073403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14177
2023-01-11T21:03:04.8073677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8073804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8074073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8074212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8074392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8074669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8074796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8075078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8075216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8075389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8075654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8075808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8076104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8076242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8076418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8076720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8077014Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8077308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8077475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8077644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8077807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8077862Z ok (2.276s)
2023-01-11T21:03:04.8077868Z 
2023-01-11T21:03:04.8078065Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8078140Z Ran 1 test in 2.276s
2023-01-11T21:03:04.8078145Z 
2023-01-11T21:03:04.8078207Z OK
2023-01-11T21:03:04.8078212Z 
2023-01-11T21:03:04.8078296Z Generating XML reports...
2023-01-11T21:03:04.8078638Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210225.xml
2023-01-11T21:03:04.8078870Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8079146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8079261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8079544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8079680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8079686Z 
2023-01-11T21:03:04.8079759Z Running tests...
2023-01-11T21:03:04.8079957Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8080212Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14228
2023-01-11T21:03:04.8080368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14229
2023-01-11T21:03:04.8080524Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14230
2023-01-11T21:03:04.8080797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8080911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8081194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8081330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8081509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8081783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8081906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8082188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8082325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8082539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8082896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8083022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8083307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8083442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8083622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8083922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8084222Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8084518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8084686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8084839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8085002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8085253Z STAGE:2023-01-11 21:02:31 14229:14229 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8085495Z STAGE:2023-01-11 21:02:31 14230:14230 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8085734Z STAGE:2023-01-11 21:02:31 14228:14228 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8085984Z STAGE:2023-01-11 21:02:31 14228:14228 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8086250Z STAGE:2023-01-11 21:02:31 14228:14228 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8086500Z STAGE:2023-01-11 21:02:31 14229:14229 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8086757Z STAGE:2023-01-11 21:02:31 14229:14229 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8086989Z STAGE:2023-01-11 21:02:31 14230:14230 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8087246Z STAGE:2023-01-11 21:02:31 14230:14230 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8087314Z ok (2.178s)
2023-01-11T21:03:04.8087320Z 
2023-01-11T21:03:04.8087517Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8087592Z Ran 1 test in 2.178s
2023-01-11T21:03:04.8087598Z 
2023-01-11T21:03:04.8087662Z OK
2023-01-11T21:03:04.8087669Z 
2023-01-11T21:03:04.8087753Z Generating XML reports...
2023-01-11T21:03:04.8088101Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210229.xml
2023-01-11T21:03:04.8088322Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8088596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8088720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8089002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8089138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8089144Z 
2023-01-11T21:03:04.8089217Z Running tests...
2023-01-11T21:03:04.8089415Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8089709Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14284
2023-01-11T21:03:04.8089894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14285
2023-01-11T21:03:04.8090039Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14286
2023-01-11T21:03:04.8090314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8090439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8090722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8090858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8091033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8091311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8091435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8091705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8091842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8092018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8092291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8092414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8092694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8092832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8093010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8093313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8093599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8093894Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8094059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8094224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8094388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8094639Z STAGE:2023-01-11 21:02:35 14284:14284 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8095047Z STAGE:2023-01-11 21:02:35 14285:14285 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:02:35 14286:14286 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:03:04.8095053Z 
2023-01-11T21:03:04.8095303Z STAGE:2023-01-11 21:02:35 14284:14284 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8095562Z STAGE:2023-01-11 21:02:35 14284:14284 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8095807Z STAGE:2023-01-11 21:02:35 14285:14285 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8096050Z STAGE:2023-01-11 21:02:35 14285:14285 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8096294Z STAGE:2023-01-11 21:02:35 14286:14286 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:03:04.8096579Z STAGE:2023-01-11 21:02:35 14286:14286 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:03:04.8096649Z ok (2.180s)
2023-01-11T21:03:04.8096686Z 
2023-01-11T21:03:04.8096885Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8096961Z Ran 1 test in 2.180s
2023-01-11T21:03:04.8096967Z 
2023-01-11T21:03:04.8097027Z OK
2023-01-11T21:03:04.8097032Z 
2023-01-11T21:03:04.8097116Z Generating XML reports...
2023-01-11T21:03:04.8097446Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210233.xml
2023-01-11T21:03:04.8097681Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8097954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8098079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8098365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8098504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8098510Z 
2023-01-11T21:03:04.8098583Z Running tests...
2023-01-11T21:03:04.8098779Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8099014Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14340
2023-01-11T21:03:04.8099159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14341
2023-01-11T21:03:04.8099316Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14342
2023-01-11T21:03:04.8099588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8099712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8099993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8100130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8100305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8100605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8100878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8100988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8101269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8101403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8101581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8101980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8102105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8102387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8102523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8102686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8102985Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8103285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8103508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8103707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8103874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8103943Z ok (2.175s)
2023-01-11T21:03:04.8103949Z 
2023-01-11T21:03:04.8104147Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8104224Z Ran 1 test in 2.176s
2023-01-11T21:03:04.8104229Z 
2023-01-11T21:03:04.8104277Z OK
2023-01-11T21:03:04.8104282Z 
2023-01-11T21:03:04.8104366Z Generating XML reports...
2023-01-11T21:03:04.8104709Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210237.xml
2023-01-11T21:03:04.8104943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8105220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8105346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8105628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8105763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8105768Z 
2023-01-11T21:03:04.8105841Z Running tests...
2023-01-11T21:03:04.8106026Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8106268Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14420
2023-01-11T21:03:04.8106424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14421
2023-01-11T21:03:04.8106583Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14422
2023-01-11T21:03:04.8106858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8106981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8107263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8107398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8107561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8107834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8107957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8108239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8108377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8108555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8108827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8108950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8109228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8109350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8109525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8109821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8110150Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8110472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8110640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8110805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8110968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8111064Z skip: CUDA is not available. (2.274s)
2023-01-11T21:03:04.8111070Z 
2023-01-11T21:03:04.8111254Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8111333Z Ran 1 test in 2.274s
2023-01-11T21:03:04.8111338Z 
2023-01-11T21:03:04.8111410Z OK (skipped=1)
2023-01-11T21:03:04.8111417Z 
2023-01-11T21:03:04.8111501Z Generating XML reports...
2023-01-11T21:03:04.8111845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210241.xml
2023-01-11T21:03:04.8112079Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8112354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8112478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8112748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8112884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8112890Z 
2023-01-11T21:03:04.8112962Z Running tests...
2023-01-11T21:03:04.8113158Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8113399Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14473
2023-01-11T21:03:04.8113559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14474
2023-01-11T21:03:04.8113716Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14475
2023-01-11T21:03:04.8113991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8114114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8114380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8114516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8114695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8114970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8115096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8115378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8115513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8115692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8115966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8116077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8116358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8116523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8116706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8117033Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8117335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8117628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8117795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8117959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8118109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8118215Z skip: Need at least 2 CUDA devices (2.277s)
2023-01-11T21:03:04.8118221Z 
2023-01-11T21:03:04.8118418Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8118496Z Ran 1 test in 2.277s
2023-01-11T21:03:04.8118501Z 
2023-01-11T21:03:04.8118574Z OK (skipped=1)
2023-01-11T21:03:04.8118579Z 
2023-01-11T21:03:04.8118665Z Generating XML reports...
2023-01-11T21:03:04.8119010Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210246.xml
2023-01-11T21:03:04.8119244Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8119506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8119631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8119913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8120052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8120057Z 
2023-01-11T21:03:04.8120132Z Running tests...
2023-01-11T21:03:04.8120325Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8120562Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14526
2023-01-11T21:03:04.8120719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14527
2023-01-11T21:03:04.8120874Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14528
2023-01-11T21:03:04.8121136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8121262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8121543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8121682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8121864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8122135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8122262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8122545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8122740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8122923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8123199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8123357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8123666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8123805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8123982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8124285Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8124584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8124864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8125030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8125196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8125360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8125430Z ok (2.282s)
2023-01-11T21:03:04.8125436Z 
2023-01-11T21:03:04.8125635Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8125710Z Ran 1 test in 2.282s
2023-01-11T21:03:04.8125715Z 
2023-01-11T21:03:04.8125777Z OK
2023-01-11T21:03:04.8125782Z 
2023-01-11T21:03:04.8125866Z Generating XML reports...
2023-01-11T21:03:04.8126196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210250.xml
2023-01-11T21:03:04.8126431Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8126706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8126834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8127118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8127255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8127260Z 
2023-01-11T21:03:04.8127334Z Running tests...
2023-01-11T21:03:04.8127531Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8127745Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14579
2023-01-11T21:03:04.8127902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14580
2023-01-11T21:03:04.8128055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14581
2023-01-11T21:03:04.8128329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8128453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8128735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8128871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8129047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8129320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8129429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8129709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8129843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8130047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8130348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8130472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8130753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8130887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8131049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8131348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8131646Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8131946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8132112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8132276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8132438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8132542Z skip: Need at least 2 CUDA devices (2.171s)
2023-01-11T21:03:04.8132548Z 
2023-01-11T21:03:04.8132744Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8132806Z Ran 1 test in 2.172s
2023-01-11T21:03:04.8132824Z 
2023-01-11T21:03:04.8132882Z OK (skipped=1)
2023-01-11T21:03:04.8132887Z 
2023-01-11T21:03:04.8132971Z Generating XML reports...
2023-01-11T21:03:04.8133317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210254.xml
2023-01-11T21:03:04.8133553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8133832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8133956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8134240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8134376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8134382Z 
2023-01-11T21:03:04.8134442Z Running tests...
2023-01-11T21:03:04.8134637Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8134896Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14632
2023-01-11T21:03:04.8135057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14633
2023-01-11T21:03:04.8135213Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14634
2023-01-11T21:03:04.8135487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8135612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8135893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8136029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8136195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:04.8136466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8136621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8136935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8137072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8137247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:03:04.8137520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8137644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8137907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8138042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8138219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:04.8138521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8138819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8139112Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:03:04.8139279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:04.8139443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:04.8139605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:03:04.8139697Z skip: Need at least 2 CUDA devices (2.181s)
2023-01-11T21:03:04.8139717Z 
2023-01-11T21:03:04.8139901Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8139975Z Ran 1 test in 2.182s
2023-01-11T21:03:04.8139981Z 
2023-01-11T21:03:04.8140053Z OK (skipped=1)
2023-01-11T21:03:04.8140061Z 
2023-01-11T21:03:04.8140145Z Generating XML reports...
2023-01-11T21:03:04.8140487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210258.xml
2023-01-11T21:03:04.8140719Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8140996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8141120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8141390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8141525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8141533Z 
2023-01-11T21:03:04.8141607Z Running tests...
2023-01-11T21:03:04.8141921Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8142274Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s)
2023-01-11T21:03:04.8142280Z 
2023-01-11T21:03:04.8142471Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8142547Z Ran 1 test in 0.002s
2023-01-11T21:03:04.8142552Z 
2023-01-11T21:03:04.8142624Z OK (skipped=1)
2023-01-11T21:03:04.8142629Z 
2023-01-11T21:03:04.8142713Z Generating XML reports...
2023-01-11T21:03:04.8143037Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210302.xml
2023-01-11T21:03:04.8143268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:03:04.8143593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:04.8143757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:04.8144041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:04.8144177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:04.8144182Z 
2023-01-11T21:03:04.8144256Z Running tests...
2023-01-11T21:03:04.8144448Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8144797Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s)
2023-01-11T21:03:04.8144803Z 
2023-01-11T21:03:04.8144985Z ----------------------------------------------------------------------
2023-01-11T21:03:04.8145065Z Ran 1 test in 0.002s
2023-01-11T21:03:04.8145070Z 
2023-01-11T21:03:04.8145141Z OK (skipped=1)
2023-01-11T21:03:04.8145146Z 
2023-01-11T21:03:04.8145231Z Generating XML reports...
2023-01-11T21:03:04.8145575Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210304.xml
2023-01-11T21:03:04.8145581Z 
2023-01-11T21:03:04.8146125Z ##[endgroup]
2023-01-11T21:03:04.8146490Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_z9rr3hxb)
2023-01-11T21:03:04.8146496Z 
2023-01-11T21:03:04.8146649Z Running distributed tests for the gloo backend with file init_method in shard 1 of 2
2023-01-11T21:03:04.8147031Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:03:04.574463]
2023-01-11T21:18:21.5078549Z 
2023-01-11T21:18:21.5079077Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2023-01-11T21:18:21.5087032Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_5uyus5i3)
2023-01-11T21:18:21.5090129Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:18:21.5130627Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_grad_as_bucket_view_false>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_ignored_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2023-01-11T21:18:21.5168360Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5168951Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5169385Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5177729Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5178393Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5178933Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5179525Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5180029Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5180640Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5181288Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5182144Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5182751Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5183409Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5184053Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5184680Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5185296Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5185936Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5186471Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5186952Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5187558Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5188220Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5188847Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5191444Z test_all_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5192190Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5192816Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5193379Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5193874Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5194319Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5194801Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5195242Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5195714Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5242306Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5242904Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5243470Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5243800Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5244098Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5244398Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5244707Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5244999Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5245416Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5245735Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5246063Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5246379Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5246705Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5267106Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5267559Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5268018Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5268457Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5268858Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5269312Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5269748Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5270177Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5270634Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5271129Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5271634Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5272085Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5272538Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5272975Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5273383Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5273776Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5274204Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5274623Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5275019Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5275421Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5275826Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5276414Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5277741Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5278343Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5278783Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5279198Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5279618Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5280013Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5280419Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5280866Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5281363Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5281916Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5282397Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5282913Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5283437Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5283934Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5284466Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5284921Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5285349Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5285809Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5286284Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5286747Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5287194Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5287670Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5288126Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5288575Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5289001Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5289442Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5289896Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5290363Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5290832Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5291434Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5292000Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5292481Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5292923Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5293317Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5293702Z test_backend_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5294058Z test_barrier (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5294428Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5294814Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5295200Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5295600Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5295996Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5296498Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5296924Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5297373Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5297784Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5298205Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5298702Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5299163Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5299647Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5300198Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5300717Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5301201Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5301932Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5302451Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5302888Z test_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5303256Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5303665Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5304071Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5304480Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5304876Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5305351Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5305878Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5306425Z test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5307016Z test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5307642Z test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5308173Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5308651Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5309119Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5309587Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5310084Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5310568Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5311051Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5311545Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5312044Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5312512Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5312993Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5313451Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5313914Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5314430Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5314947Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5315528Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5316078Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5316770Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5317484Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5318130Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5318830Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5319616Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5320423Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5321199Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5322097Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5322943Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5323762Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5324509Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5325225Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5325870Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5326456Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5326962Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5327487Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5353634Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5354120Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5354665Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5355234Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5355803Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5356354Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5356878Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5357365Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5357875Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5358406Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5358942Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5359450Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5359947Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5360480Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5361011Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5361515Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5362100Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5362600Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5363267Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5363744Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5364310Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5364928Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5365475Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5365978Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5366458Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5366947Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5367460Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5367977Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5368456Z test_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5368893Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5369369Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5369826Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5370296Z test_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5370733Z test_gather_object (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5371214Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5371687Z test_get_backend (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5372122Z test_get_future (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5372563Z test_get_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5373047Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5373517Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5374017Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5374482Z test_irecv (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5374934Z test_isend (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5375392Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5375912Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5376436Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5376992Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5377563Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5378086Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5378615Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5379141Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5379718Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5380297Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5380851Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5381390Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5381883Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5382191Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5382469Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5382764Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5383115Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5383467Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5383815Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5384228Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5384610Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5384940Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5385265Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5385576Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5385882Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5386205Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5386539Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5386897Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5387266Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5387626Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5387937Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5388220Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5388516Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5388815Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5389089Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5389370Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5389654Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5389939Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5390199Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5390471Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5390746Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5391013Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5391308Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5391606Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5391870Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5392146Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5392430Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5392715Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5392971Z test_scatter (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5393240Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5393518Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5393780Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5394063Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5394354Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5394623Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5394905Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5395179Z test_send_recv (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5395455Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5395754Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5396082Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5396396Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5396673Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5397005Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5397349Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5397642Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5397933Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5398236Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5398550Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5398840Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5399139Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5399438Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5399719Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5400004Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5400317Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5400637Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5400966Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5401476Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5402049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5402373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5402806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5403155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5403325Z 
2023-01-11T21:18:21.5403400Z Running tests...
2023-01-11T21:18:21.5403696Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5404160Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14715
2023-01-11T21:18:21.5404619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14716
2023-01-11T21:18:21.5404942Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14717
2023-01-11T21:18:21.5405399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5405730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5406156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5406488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5406828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5407305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5407632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5408044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5408388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5408723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5409179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5409506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5409965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5410340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5410661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5411163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5411685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5412204Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5412577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5412927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5413267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5413537Z skip: Need at least 2 CUDA devices (2.176s)
2023-01-11T21:18:21.5413670Z 
2023-01-11T21:18:21.5413873Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5414117Z Ran 1 test in 2.176s
2023-01-11T21:18:21.5414230Z 
2023-01-11T21:18:21.5414307Z OK (skipped=1)
2023-01-11T21:18:21.5414449Z 
2023-01-11T21:18:21.5414555Z Generating XML reports...
2023-01-11T21:18:21.5415018Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210307.xml
2023-01-11T21:18:21.5415518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5415971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5416308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5416739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5417083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5417233Z 
2023-01-11T21:18:21.5417306Z Running tests...
2023-01-11T21:18:21.5417611Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5417985Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2023-01-11T21:18:21.5418211Z 
2023-01-11T21:18:21.5418406Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5418635Z Ran 1 test in 0.003s
2023-01-11T21:18:21.5418747Z 
2023-01-11T21:18:21.5418818Z OK (skipped=1)
2023-01-11T21:18:21.5418926Z 
2023-01-11T21:18:21.5419009Z Generating XML reports...
2023-01-11T21:18:21.5419458Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210311.xml
2023-01-11T21:18:21.5419957Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5420422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5420751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5421168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5421514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5421812Z 
2023-01-11T21:18:21.5421890Z Running tests...
2023-01-11T21:18:21.5422185Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5422589Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14778
2023-01-11T21:18:21.5423066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14779
2023-01-11T21:18:21.5423433Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14780
2023-01-11T21:18:21.5423878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5424206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5424636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5424970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5425305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5425776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5426101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5426518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5426866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5427203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5427668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5427982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5428409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5428750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5429068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5429568Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5430088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5430600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5430970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5431325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5431674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5431926Z ok (2.274s)
2023-01-11T21:18:21.5432016Z 
2023-01-11T21:18:21.5432219Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5432462Z Ran 1 test in 2.274s
2023-01-11T21:18:21.5432572Z 
2023-01-11T21:18:21.5432634Z OK
2023-01-11T21:18:21.5432712Z 
2023-01-11T21:18:21.5432801Z Generating XML reports...
2023-01-11T21:18:21.5433261Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210313.xml
2023-01-11T21:18:21.5433757Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5434204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5434534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5434963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5435306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5435519Z 
2023-01-11T21:18:21.5435579Z Running tests...
2023-01-11T21:18:21.5435886Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5436794Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s)
2023-01-11T21:18:21.5437326Z 
2023-01-11T21:18:21.5437527Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5437771Z Ran 1 test in 0.254s
2023-01-11T21:18:21.5437883Z 
2023-01-11T21:18:21.5437942Z OK (skipped=1)
2023-01-11T21:18:21.5438049Z 
2023-01-11T21:18:21.5438135Z Generating XML reports...
2023-01-11T21:18:21.5438594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210317.xml
2023-01-11T21:18:21.5439092Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5439546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5439877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5440308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5440645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5440811Z 
2023-01-11T21:18:21.5440883Z Running tests...
2023-01-11T21:18:21.5441184Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5441684Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14841
2023-01-11T21:18:21.5442104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14842
2023-01-11T21:18:21.5442437Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14843
2023-01-11T21:18:21.5442893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5443212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5443642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5443986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5444324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5444780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5445108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5445536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5445879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5446203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5446672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5447001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5447418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5447761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5448092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5448626Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5449160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5449681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5450066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5450416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5450749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5451101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.5451458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.5451801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.5452059Z ok (2.373s)
2023-01-11T21:18:21.5452165Z 
2023-01-11T21:18:21.5452369Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5452608Z Ran 1 test in 2.373s
2023-01-11T21:18:21.5452708Z 
2023-01-11T21:18:21.5452769Z OK
2023-01-11T21:18:21.5452862Z 
2023-01-11T21:18:21.5452947Z Generating XML reports...
2023-01-11T21:18:21.5453406Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210319.xml
2023-01-11T21:18:21.5453889Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5454350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5454681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5455117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5455452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5455613Z 
2023-01-11T21:18:21.5455686Z Running tests...
2023-01-11T21:18:21.5455985Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5456406Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14894
2023-01-11T21:18:21.5456839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14895
2023-01-11T21:18:21.5457171Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14896
2023-01-11T21:18:21.5457618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5457934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5458361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5458706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5459031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5459501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5459830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5460258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5460588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5460922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5461422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5461946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5462372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5462719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5463055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5463536Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5464057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5464576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5464968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5465307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5465648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5465999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.5466358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.5466704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.5466959Z ok (2.374s)
2023-01-11T21:18:21.5467063Z 
2023-01-11T21:18:21.5467263Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5467490Z Ran 1 test in 2.375s
2023-01-11T21:18:21.5467604Z 
2023-01-11T21:18:21.5467666Z OK
2023-01-11T21:18:21.5467757Z 
2023-01-11T21:18:21.5467841Z Generating XML reports...
2023-01-11T21:18:21.5468301Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210323.xml
2023-01-11T21:18:21.5468790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5469251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5469579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5469996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5470340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5470505Z 
2023-01-11T21:18:21.5470578Z Running tests...
2023-01-11T21:18:21.5470880Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5471305Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14947
2023-01-11T21:18:21.5471735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14948
2023-01-11T21:18:21.5472067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14949
2023-01-11T21:18:21.5472510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5472839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5473267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5473614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5473940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5474453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5474809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5475239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5475570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5475906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5476371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5476685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5477112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5477458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5477792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5478273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5478790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5479306Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5479690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5480022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5480360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5480642Z skip: CUDA is not available. (2.178s)
2023-01-11T21:18:21.5480774Z 
2023-01-11T21:18:21.5480961Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5481204Z Ran 1 test in 2.178s
2023-01-11T21:18:21.5481316Z 
2023-01-11T21:18:21.5481389Z OK (skipped=1)
2023-01-11T21:18:21.5481494Z 
2023-01-11T21:18:21.5481645Z Generating XML reports...
2023-01-11T21:18:21.5482103Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210327.xml
2023-01-11T21:18:21.5482602Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5483065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5483384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5483813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5484160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5484325Z 
2023-01-11T21:18:21.5484397Z Running tests...
2023-01-11T21:18:21.5484686Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5485128Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15000
2023-01-11T21:18:21.5485567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15001
2023-01-11T21:18:21.5485902Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15002
2023-01-11T21:18:21.5486343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5486672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5487149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5487509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5487849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5488321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5488652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5489071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5489416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5489751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5490233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5490736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5491063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5491488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5491826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5492157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5492654Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5493176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5493551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5493904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5494247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5494513Z skip: CUDA is not available. (2.183s)
2023-01-11T21:18:21.5494644Z 
2023-01-11T21:18:21.5494844Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5495084Z Ran 1 test in 2.183s
2023-01-11T21:18:21.5495196Z 
2023-01-11T21:18:21.5495267Z OK (skipped=1)
2023-01-11T21:18:21.5495361Z 
2023-01-11T21:18:21.5495450Z Generating XML reports...
2023-01-11T21:18:21.5495914Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210331.xml
2023-01-11T21:18:21.5496411Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5496866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5497198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5497629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5497975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5498139Z 
2023-01-11T21:18:21.5498201Z Running tests...
2023-01-11T21:18:21.5498507Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5498960Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15053
2023-01-11T21:18:21.5499407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15054
2023-01-11T21:18:21.5499761Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15055
2023-01-11T21:18:21.5500242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5500574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5500993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5501343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5501807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5502294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5502609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5503042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5503393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5503733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5504189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5504522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5504951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5505282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5505616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5506114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5506643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5507145Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5507532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5507883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5508234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5508502Z skip: CUDA is not available. (2.175s)
2023-01-11T21:18:21.5508631Z 
2023-01-11T21:18:21.5508834Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5509072Z Ran 1 test in 2.175s
2023-01-11T21:18:21.5509186Z 
2023-01-11T21:18:21.5509244Z OK (skipped=1)
2023-01-11T21:18:21.5509352Z 
2023-01-11T21:18:21.5509435Z Generating XML reports...
2023-01-11T21:18:21.5509899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210335.xml
2023-01-11T21:18:21.5510398Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5510850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5511183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5511611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5511942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5512107Z 
2023-01-11T21:18:21.5512183Z Running tests...
2023-01-11T21:18:21.5512486Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5513061Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15106
2023-01-11T21:18:21.5513512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15107
2023-01-11T21:18:21.5513848Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15108
2023-01-11T21:18:21.5514303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5514634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5515052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5515399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5515741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5516206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5516535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5516963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5517307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5517627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5518103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5518432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5518850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5519194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5519532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5520029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5520537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5521061Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5521454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5521894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5522228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5522507Z skip: CUDA is not available. (2.172s)
2023-01-11T21:18:21.5522641Z 
2023-01-11T21:18:21.5522848Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5523073Z Ran 1 test in 2.172s
2023-01-11T21:18:21.5523186Z 
2023-01-11T21:18:21.5523257Z OK (skipped=1)
2023-01-11T21:18:21.5523364Z 
2023-01-11T21:18:21.5523448Z Generating XML reports...
2023-01-11T21:18:21.5524031Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210339.xml
2023-01-11T21:18:21.5524522Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5524987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5525323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5525808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5526207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5526375Z 
2023-01-11T21:18:21.5526449Z Running tests...
2023-01-11T21:18:21.5526754Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5527202Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15159
2023-01-11T21:18:21.5527661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15160
2023-01-11T21:18:21.5528000Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15161
2023-01-11T21:18:21.5528458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5528777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5529207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5529555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5529892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5530369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5530700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5531128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5531462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5531798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5532304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5532806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5533123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5533554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5533901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5534225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5534725Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5535254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5535650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5535987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5536329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5536607Z skip: CUDA is not available. (2.180s)
2023-01-11T21:18:21.5536738Z 
2023-01-11T21:18:21.5536939Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5537166Z Ran 1 test in 2.180s
2023-01-11T21:18:21.5537277Z 
2023-01-11T21:18:21.5537347Z OK (skipped=1)
2023-01-11T21:18:21.5537455Z 
2023-01-11T21:18:21.5537540Z Generating XML reports...
2023-01-11T21:18:21.5537993Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210343.xml
2023-01-11T21:18:21.5538540Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5539039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5539380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5539801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5540152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5540314Z 
2023-01-11T21:18:21.5540388Z Running tests...
2023-01-11T21:18:21.5540684Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5541131Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15212
2023-01-11T21:18:21.5541575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15213
2023-01-11T21:18:21.5542075Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15214
2023-01-11T21:18:21.5542533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5542866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5543305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5543639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5543982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5544457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5544787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5545210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5545569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5545908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5546389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5546704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5547137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5547482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5547805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5548301Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5548840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5549362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5549736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5550086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5550438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5550722Z skip: CUDA is not available. (2.276s)
2023-01-11T21:18:21.5550842Z 
2023-01-11T21:18:21.5551047Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5551289Z Ran 1 test in 2.276s
2023-01-11T21:18:21.5551488Z 
2023-01-11T21:18:21.5551560Z OK (skipped=1)
2023-01-11T21:18:21.5551674Z 
2023-01-11T21:18:21.5551746Z Generating XML reports...
2023-01-11T21:18:21.5552243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210347.xml
2023-01-11T21:18:21.5552744Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5553213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5553532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5553966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5554313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5554477Z 
2023-01-11T21:18:21.5554539Z Running tests...
2023-01-11T21:18:21.5554845Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5555310Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15265
2023-01-11T21:18:21.5555767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15266
2023-01-11T21:18:21.5556089Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15267
2023-01-11T21:18:21.5556544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5556876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5557308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5557648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5557987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5558457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5558772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5559203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5559555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5559895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5560360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5560688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5561115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5561447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5561860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5562359Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5562881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5563388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5563776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5564128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5564521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5564790Z skip: CUDA is not available. (2.283s)
2023-01-11T21:18:21.5564919Z 
2023-01-11T21:18:21.5565150Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5565395Z Ran 1 test in 2.284s
2023-01-11T21:18:21.5565509Z 
2023-01-11T21:18:21.5565568Z OK (skipped=1)
2023-01-11T21:18:21.5565678Z 
2023-01-11T21:18:21.5565764Z Generating XML reports...
2023-01-11T21:18:21.5566222Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210351.xml
2023-01-11T21:18:21.5566715Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5567174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5567513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5567951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5568304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5568455Z 
2023-01-11T21:18:21.5568530Z Running tests...
2023-01-11T21:18:21.5568836Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5569757Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2023-01-11T21:18:21.5570250Z 
2023-01-11T21:18:21.5570450Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5570676Z Ran 1 test in 0.255s
2023-01-11T21:18:21.5570793Z 
2023-01-11T21:18:21.5570870Z OK (skipped=1)
2023-01-11T21:18:21.5570979Z 
2023-01-11T21:18:21.5571068Z Generating XML reports...
2023-01-11T21:18:21.5571515Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210355.xml
2023-01-11T21:18:21.5572015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5572481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5572810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5573225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5573574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5573737Z 
2023-01-11T21:18:21.5573810Z Running tests...
2023-01-11T21:18:21.5574107Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5574538Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15328
2023-01-11T21:18:21.5574967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15329
2023-01-11T21:18:21.5575304Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15330
2023-01-11T21:18:21.5575746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5576075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5576505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5576849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5577174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5577725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5578053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5578471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5578814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5579145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5579617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5579931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5580359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5580707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5581044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5581531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5582198Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5582719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5583098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5583451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5583798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5584051Z ok (2.172s)
2023-01-11T21:18:21.5584143Z 
2023-01-11T21:18:21.5584349Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5584595Z Ran 1 test in 2.173s
2023-01-11T21:18:21.5584707Z 
2023-01-11T21:18:21.5584771Z OK
2023-01-11T21:18:21.5584864Z 
2023-01-11T21:18:21.5584937Z Generating XML reports...
2023-01-11T21:18:21.5585403Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210358.xml
2023-01-11T21:18:21.5585904Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5586372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5586689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5587123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5587476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5587642Z 
2023-01-11T21:18:21.5587718Z Running tests...
2023-01-11T21:18:21.5588012Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5588930Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.262s)
2023-01-11T21:18:21.5589432Z 
2023-01-11T21:18:21.5589629Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5589872Z Ran 1 test in 0.263s
2023-01-11T21:18:21.5589970Z 
2023-01-11T21:18:21.5590114Z OK (skipped=1)
2023-01-11T21:18:21.5590221Z 
2023-01-11T21:18:21.5590305Z Generating XML reports...
2023-01-11T21:18:21.5590801Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210402.xml
2023-01-11T21:18:21.5591282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5591750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5592083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5592516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5592853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5593015Z 
2023-01-11T21:18:21.5593089Z Running tests...
2023-01-11T21:18:21.5593399Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5593801Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15391
2023-01-11T21:18:21.5594225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15392
2023-01-11T21:18:21.5594557Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15393
2023-01-11T21:18:21.5595012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5595329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5595757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5596105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5596443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5596905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5597240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5597669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5597999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5598337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5598810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5599136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5599548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5599897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5600232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5600728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5601237Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5601826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5602220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5602554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5602900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5603232Z skip: Need at least 3 CUDA devices (2.172s)
2023-01-11T21:18:21.5603369Z 
2023-01-11T21:18:21.5603605Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5603837Z Ran 1 test in 2.173s
2023-01-11T21:18:21.5603949Z 
2023-01-11T21:18:21.5604022Z OK (skipped=1)
2023-01-11T21:18:21.5604130Z 
2023-01-11T21:18:21.5604215Z Generating XML reports...
2023-01-11T21:18:21.5604661Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210404.xml
2023-01-11T21:18:21.5605157Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5605619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5605954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5606369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5606717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5606881Z 
2023-01-11T21:18:21.5606957Z Running tests...
2023-01-11T21:18:21.5607262Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5607599Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2023-01-11T21:18:21.5607808Z 
2023-01-11T21:18:21.5608003Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5608245Z Ran 1 test in 0.002s
2023-01-11T21:18:21.5608359Z 
2023-01-11T21:18:21.5608419Z OK (skipped=1)
2023-01-11T21:18:21.5608526Z 
2023-01-11T21:18:21.5608611Z Generating XML reports...
2023-01-11T21:18:21.5609070Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210408.xml
2023-01-11T21:18:21.5609560Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5610015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5610347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5610781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5611115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5611280Z 
2023-01-11T21:18:21.5611353Z Running tests...
2023-01-11T21:18:21.5611653Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5611969Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5612354Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15454
2023-01-11T21:18:21.5612755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15455
2023-01-11T21:18:21.5613090Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15456
2023-01-11T21:18:21.5613534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5613864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5614304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5614654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5614974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5615448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5615778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5616267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5616630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5616967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5617441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5617758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5618186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5618530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5618868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5619354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5619887Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5620407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5620791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5621129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5621476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5622003Z ok (2.173s)
2023-01-11T21:18:21.5622109Z 
2023-01-11T21:18:21.5622302Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5622547Z Ran 1 test in 2.174s
2023-01-11T21:18:21.5622661Z 
2023-01-11T21:18:21.5622724Z OK
2023-01-11T21:18:21.5622815Z 
2023-01-11T21:18:21.5622901Z Generating XML reports...
2023-01-11T21:18:21.5623352Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210409.xml
2023-01-11T21:18:21.5623850Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5624317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5624633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5625064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5625409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5625572Z 
2023-01-11T21:18:21.5625648Z Running tests...
2023-01-11T21:18:21.5625945Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5626277Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5626697Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15507
2023-01-11T21:18:21.5627081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15508
2023-01-11T21:18:21.5627416Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15509
2023-01-11T21:18:21.5627868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5628203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5628625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5629042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5629383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5629896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5630217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5630652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5630994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5631316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5631786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5632114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5632544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5632874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5633211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5633706Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5634214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5634732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5635123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5635473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5635810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5636172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.5636533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.5636889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.5637133Z ok (2.172s)
2023-01-11T21:18:21.5637237Z 
2023-01-11T21:18:21.5637440Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5637682Z Ran 1 test in 2.172s
2023-01-11T21:18:21.5637796Z 
2023-01-11T21:18:21.5637845Z OK
2023-01-11T21:18:21.5637936Z 
2023-01-11T21:18:21.5638020Z Generating XML reports...
2023-01-11T21:18:21.5638483Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210413.xml
2023-01-11T21:18:21.5638988Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5639444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5639776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5640207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5640540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5640702Z 
2023-01-11T21:18:21.5640777Z Running tests...
2023-01-11T21:18:21.5641079Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5641425Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5641943Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15560
2023-01-11T21:18:21.5642435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15561
2023-01-11T21:18:21.5642764Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15562
2023-01-11T21:18:21.5643220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5643538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5643970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5644315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5644638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5645114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5645447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5645878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5646209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5646542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5647008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5647326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5647756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5648101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5648435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5648914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5649434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5649951Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5650334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5650671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5651018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5651372Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.5651719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.5652082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.5652345Z ok (2.175s)
2023-01-11T21:18:21.5652448Z 
2023-01-11T21:18:21.5652651Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5652879Z Ran 1 test in 2.175s
2023-01-11T21:18:21.5652993Z 
2023-01-11T21:18:21.5653054Z OK
2023-01-11T21:18:21.5653147Z 
2023-01-11T21:18:21.5653238Z Generating XML reports...
2023-01-11T21:18:21.5653689Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210417.xml
2023-01-11T21:18:21.5654195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5654647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5655015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5655477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5655829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5655981Z 
2023-01-11T21:18:21.5656056Z Running tests...
2023-01-11T21:18:21.5656360Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5656697Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:18:21.5657097Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15613
2023-01-11T21:18:21.5657497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15614
2023-01-11T21:18:21.5657832Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15615
2023-01-11T21:18:21.5658290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5658610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5659043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5659390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5659726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5660185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5660518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5660950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5661285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5661630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5662221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5662550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5662969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5663316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5663652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5664138Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5664669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5665191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5665578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5665916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5666259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5666509Z ok (2.180s)
2023-01-11T21:18:21.5666614Z 
2023-01-11T21:18:21.5666813Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5667037Z Ran 1 test in 2.180s
2023-01-11T21:18:21.5667148Z 
2023-01-11T21:18:21.5667210Z OK
2023-01-11T21:18:21.5667300Z 
2023-01-11T21:18:21.5667384Z Generating XML reports...
2023-01-11T21:18:21.5667892Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210421.xml
2023-01-11T21:18:21.5668424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5668893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5669225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5669640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5669988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5670151Z 
2023-01-11T21:18:21.5670224Z Running tests...
2023-01-11T21:18:21.5670513Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5670897Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15666
2023-01-11T21:18:21.5671299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15667
2023-01-11T21:18:21.5671637Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15668
2023-01-11T21:18:21.5672077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5672406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5672834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5673181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5673503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5673974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5674302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5674719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5675068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5675399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5675866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5676180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5676606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5676949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5677272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5677767Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5678290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5678802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5678968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5679206Z STAGE:2023-01-11 21:04:27 15666:15666 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5679369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5679532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5679816Z STAGE:2023-01-11 21:04:27 15668:15668 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5680083Z STAGE:2023-01-11 21:04:27 15667:15667 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5680332Z STAGE:2023-01-11 21:04:27 15667:15667 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5680755Z STAGE:2023-01-11 21:04:27 15667:15667 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:04:27 15668:15668 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5680761Z 
2023-01-11T21:18:21.5681011Z STAGE:2023-01-11 21:04:27 15666:15666 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5681445Z STAGE:2023-01-11 21:04:27 15668:15668 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:04:27 15666:15666 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5681454Z 
2023-01-11T21:18:21.5681761Z STAGE:2023-01-11 21:04:27 15667:15667 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5682005Z STAGE:2023-01-11 21:04:27 15668:15668 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5682229Z STAGE:2023-01-11 21:04:27 15666:15666 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5682478Z STAGE:2023-01-11 21:04:27 15667:15667 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5682734Z STAGE:2023-01-11 21:04:27 15667:15667 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5682979Z STAGE:2023-01-11 21:04:27 15668:15668 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5683223Z STAGE:2023-01-11 21:04:27 15666:15666 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5683480Z STAGE:2023-01-11 21:04:27 15668:15668 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5683743Z STAGE:2023-01-11 21:04:27 15666:15666 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5683990Z STAGE:2023-01-11 21:04:27 15667:15667 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5684389Z STAGE:2023-01-11 21:04:27 15666:15666 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:04:27 15668:15668 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5684396Z 
2023-01-11T21:18:21.5684629Z STAGE:2023-01-11 21:04:27 15666:15666 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5684870Z STAGE:2023-01-11 21:04:27 15667:15667 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5685108Z STAGE:2023-01-11 21:04:27 15668:15668 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5685547Z STAGE:2023-01-11 21:04:27 15667:15667 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:04:27 15666:15666 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5685553Z 
2023-01-11T21:18:21.5685811Z STAGE:2023-01-11 21:04:27 15668:15668 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5685879Z ok (2.375s)
2023-01-11T21:18:21.5685885Z 
2023-01-11T21:18:21.5686085Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5686161Z Ran 1 test in 2.375s
2023-01-11T21:18:21.5686166Z 
2023-01-11T21:18:21.5686226Z OK
2023-01-11T21:18:21.5686231Z 
2023-01-11T21:18:21.5686317Z Generating XML reports...
2023-01-11T21:18:21.5686648Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210425.xml
2023-01-11T21:18:21.5686883Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5687197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5687350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5687634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5687770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5687776Z 
2023-01-11T21:18:21.5687850Z Running tests...
2023-01-11T21:18:21.5688050Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5688284Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15722
2023-01-11T21:18:21.5688441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15723
2023-01-11T21:18:21.5688593Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15724
2023-01-11T21:18:21.5688870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5688996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5689276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5689410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5689586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5689856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5689967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5690248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5690384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5690562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5690831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5690954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5691232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5691368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5691530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5691828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5692129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5692426Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5692590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5692756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5692918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5693168Z STAGE:2023-01-11 21:04:31 15722:15722 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5693414Z STAGE:2023-01-11 21:04:31 15723:15723 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5693651Z STAGE:2023-01-11 21:04:31 15724:15724 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5694287Z STAGE:2023-01-11 21:04:31 15724:15724 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:04:31 15722:15722 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:04:31 15723:15723 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5694306Z 
2023-01-11T21:18:21.5694311Z 
2023-01-11T21:18:21.5694913Z STAGE:2023-01-11 21:04:31 15723:15723 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:04:31 15724:15724 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:04:31 15722:15722 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5694919Z 
2023-01-11T21:18:21.5694924Z 
2023-01-11T21:18:21.5695321Z STAGE:2023-01-11 21:04:31 15723:15723 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:04:31 15724:15724 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5695330Z 
2023-01-11T21:18:21.5695568Z STAGE:2023-01-11 21:04:31 15722:15722 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5695803Z STAGE:2023-01-11 21:04:31 15723:15723 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5696059Z STAGE:2023-01-11 21:04:31 15723:15723 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5696467Z STAGE:2023-01-11 21:04:31 15724:15724 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:04:31 15722:15722 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5696472Z 
2023-01-11T21:18:21.5696905Z STAGE:2023-01-11 21:04:31 15722:15722 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:04:31 15724:15724 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5696911Z 
2023-01-11T21:18:21.5697158Z STAGE:2023-01-11 21:04:31 15723:15723 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5697399Z STAGE:2023-01-11 21:04:31 15722:15722 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5697636Z STAGE:2023-01-11 21:04:31 15724:15724 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5697882Z STAGE:2023-01-11 21:04:31 15722:15722 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5698287Z STAGE:2023-01-11 21:04:31 15723:15723 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:04:31 15724:15724 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5698293Z 
2023-01-11T21:18:21.5698546Z STAGE:2023-01-11 21:04:31 15722:15722 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5698968Z STAGE:2023-01-11 21:04:31 15723:15723 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:04:31 15724:15724 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5698977Z 
2023-01-11T21:18:21.5699550Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5699701Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5700247Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5700394Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5700972Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5701151Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5701223Z ok (2.279s)
2023-01-11T21:18:21.5701229Z 
2023-01-11T21:18:21.5701429Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5701504Z Ran 1 test in 2.280s
2023-01-11T21:18:21.5701510Z 
2023-01-11T21:18:21.5701568Z OK
2023-01-11T21:18:21.5701574Z 
2023-01-11T21:18:21.5701747Z Generating XML reports...
2023-01-11T21:18:21.5702080Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210429.xml
2023-01-11T21:18:21.5702312Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5702593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5702721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5703002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5703140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5703146Z 
2023-01-11T21:18:21.5703220Z Running tests...
2023-01-11T21:18:21.5703417Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5703666Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15778
2023-01-11T21:18:21.5703812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15779
2023-01-11T21:18:21.5703970Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15780
2023-01-11T21:18:21.5704249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5704376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5704657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5704794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5704973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5705245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5705368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5705635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5705774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5705959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5706259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5706531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5706655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5706937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5707071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5707231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5707574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5707905Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5708073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5708238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5708402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5708579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.5708752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.5708926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.5709211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5709510Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5709803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5710052Z STAGE:2023-01-11 21:04:35 15779:15779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5710293Z STAGE:2023-01-11 21:04:35 15780:15780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5710529Z STAGE:2023-01-11 21:04:35 15778:15778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5710777Z STAGE:2023-01-11 21:04:35 15779:15779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5711189Z STAGE:2023-01-11 21:04:35 15780:15780 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:04:35 15778:15778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5711198Z 
2023-01-11T21:18:21.5711455Z STAGE:2023-01-11 21:04:35 15779:15779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5711886Z STAGE:2023-01-11 21:04:35 15778:15778 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:04:35 15780:15780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5711892Z 
2023-01-11T21:18:21.5712134Z STAGE:2023-01-11 21:04:35 15779:15779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5712517Z STAGE:2023-01-11 21:04:35 15778:15778 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:04:35 15780:15780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5712536Z 
2023-01-11T21:18:21.5712770Z STAGE:2023-01-11 21:04:35 15780:15780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5713015Z STAGE:2023-01-11 21:04:35 15779:15779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5713255Z STAGE:2023-01-11 21:04:35 15778:15778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5713512Z STAGE:2023-01-11 21:04:35 15780:15780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5713768Z STAGE:2023-01-11 21:04:35 15779:15779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5714022Z STAGE:2023-01-11 21:04:35 15778:15778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5714261Z STAGE:2023-01-11 21:04:35 15780:15780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5714498Z STAGE:2023-01-11 21:04:35 15779:15779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5714769Z STAGE:2023-01-11 21:04:35 15778:15778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5715200Z STAGE:2023-01-11 21:04:35 15779:15779 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:04:35 15780:15780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5715447Z STAGE:2023-01-11 21:04:35 15778:15778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5715453Z 
2023-01-11T21:18:21.5715704Z STAGE:2023-01-11 21:04:35 15779:15779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5715955Z STAGE:2023-01-11 21:04:35 15780:15780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5716205Z STAGE:2023-01-11 21:04:35 15778:15778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5716776Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5716928Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5717480Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5717630Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5718190Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5718339Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5718406Z ok (2.285s)
2023-01-11T21:18:21.5718413Z 
2023-01-11T21:18:21.5718598Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5718672Z Ran 1 test in 2.286s
2023-01-11T21:18:21.5718679Z 
2023-01-11T21:18:21.5718738Z OK
2023-01-11T21:18:21.5718744Z 
2023-01-11T21:18:21.5718827Z Generating XML reports...
2023-01-11T21:18:21.5719167Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210434.xml
2023-01-11T21:18:21.5719402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5719677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5719802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5725772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5725907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5725914Z 
2023-01-11T21:18:21.5725990Z Running tests...
2023-01-11T21:18:21.5726188Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5726435Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15843
2023-01-11T21:18:21.5726595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15844
2023-01-11T21:18:21.5726751Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15845
2023-01-11T21:18:21.5727027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5727248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5727553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5727695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5727874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5728150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5728276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5728586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5728722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5728901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5729177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5729290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5729573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5729708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5729886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5730187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5730484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5730780Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5730950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5731114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5731265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5731441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.5731617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.5731788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.5732083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5732384Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5732639Z STAGE:2023-01-11 21:04:40 15844:15844 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5732882Z STAGE:2023-01-11 21:04:40 15845:15845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5733295Z STAGE:2023-01-11 21:04:40 15845:15845 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:04:40 15844:15844 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5733301Z 
2023-01-11T21:18:21.5733730Z STAGE:2023-01-11 21:04:40 15844:15844 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:04:40 15845:15845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5733736Z 
2023-01-11T21:18:21.5734129Z STAGE:2023-01-11 21:04:40 15844:15844 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:04:40 15845:15845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5734166Z 
2023-01-11T21:18:21.5734429Z STAGE:2023-01-11 21:04:40 15845:15845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5734673Z STAGE:2023-01-11 21:04:40 15844:15844 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5734929Z STAGE:2023-01-11 21:04:40 15845:15845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5735184Z STAGE:2023-01-11 21:04:40 15844:15844 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5735424Z STAGE:2023-01-11 21:04:40 15845:15845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5735664Z STAGE:2023-01-11 21:04:40 15844:15844 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5735910Z STAGE:2023-01-11 21:04:40 15844:15844 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5736156Z STAGE:2023-01-11 21:04:40 15845:15845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5736413Z STAGE:2023-01-11 21:04:40 15844:15844 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5736657Z STAGE:2023-01-11 21:04:40 15845:15845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5736955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5737529Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5737679Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5738233Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5738379Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5738447Z ok (2.380s)
2023-01-11T21:18:21.5738452Z 
2023-01-11T21:18:21.5738650Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5738725Z Ran 1 test in 2.380s
2023-01-11T21:18:21.5738730Z 
2023-01-11T21:18:21.5738790Z OK
2023-01-11T21:18:21.5738795Z 
2023-01-11T21:18:21.5738868Z Generating XML reports...
2023-01-11T21:18:21.5739209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210438.xml
2023-01-11T21:18:21.5739439Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5739715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5739841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5740123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5740259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5740265Z 
2023-01-11T21:18:21.5740337Z Running tests...
2023-01-11T21:18:21.5740533Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5740768Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15904
2023-01-11T21:18:21.5740925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15905
2023-01-11T21:18:21.5741113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15906
2023-01-11T21:18:21.5741416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5741539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5741937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5742074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5742251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5742522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5742631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5742916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5743053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5743236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5743508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5743632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5743911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5744047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5744209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5744509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5744809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5745102Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5745269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5745433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5745595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5745846Z STAGE:2023-01-11 21:04:44 15904:15904 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5746087Z STAGE:2023-01-11 21:04:44 15905:15905 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5746318Z STAGE:2023-01-11 21:04:44 15906:15906 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5746900Z STAGE:2023-01-11 21:04:44 15906:15906 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:04:44 15905:15905 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:04:44 15904:15904 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5746919Z 
2023-01-11T21:18:21.5746923Z 
2023-01-11T21:18:21.5747511Z STAGE:2023-01-11 21:04:44 15906:15906 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:04:44 15904:15904 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:04:44 15905:15905 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5747528Z 
2023-01-11T21:18:21.5747532Z 
2023-01-11T21:18:21.5747769Z STAGE:2023-01-11 21:04:44 15906:15906 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5748151Z STAGE:2023-01-11 21:04:44 15904:15904 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:04:44 15905:15905 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5748240Z 
2023-01-11T21:18:21.5748513Z STAGE:2023-01-11 21:04:44 15904:15904 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5748759Z STAGE:2023-01-11 21:04:44 15906:15906 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5749005Z STAGE:2023-01-11 21:04:44 15905:15905 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5749261Z STAGE:2023-01-11 21:04:44 15904:15904 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5749517Z STAGE:2023-01-11 21:04:44 15906:15906 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5749771Z STAGE:2023-01-11 21:04:44 15905:15905 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5750017Z STAGE:2023-01-11 21:04:44 15904:15904 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5750258Z STAGE:2023-01-11 21:04:44 15906:15906 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5750481Z STAGE:2023-01-11 21:04:44 15905:15905 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5750723Z STAGE:2023-01-11 21:04:44 15906:15906 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5750962Z STAGE:2023-01-11 21:04:44 15904:15904 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5751198Z STAGE:2023-01-11 21:04:44 15905:15905 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5751454Z STAGE:2023-01-11 21:04:44 15906:15906 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5751708Z STAGE:2023-01-11 21:04:44 15904:15904 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5751960Z STAGE:2023-01-11 21:04:44 15905:15905 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5752525Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5752679Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5753232Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5753384Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5753927Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5754076Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5754145Z ok (2.273s)
2023-01-11T21:18:21.5754150Z 
2023-01-11T21:18:21.5754351Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5754429Z Ran 1 test in 2.273s
2023-01-11T21:18:21.5754435Z 
2023-01-11T21:18:21.5754495Z OK
2023-01-11T21:18:21.5754500Z 
2023-01-11T21:18:21.5754583Z Generating XML reports...
2023-01-11T21:18:21.5754925Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210442.xml
2023-01-11T21:18:21.5755155Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5755449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5755599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5755884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5756019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5756025Z 
2023-01-11T21:18:21.5756098Z Running tests...
2023-01-11T21:18:21.5756294Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5756542Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15960
2023-01-11T21:18:21.5756698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15961
2023-01-11T21:18:21.5756851Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15962
2023-01-11T21:18:21.5757114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5757238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5757520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5757657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5757835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5758136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5758410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5758531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5758799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5758934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5759108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5759378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5759500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5759781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5759914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5760091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5760393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5760680Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5760843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5761006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5761168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5761414Z STAGE:2023-01-11 21:04:48 15960:15960 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5761734Z STAGE:2023-01-11 21:04:48 15961:15961 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5761975Z STAGE:2023-01-11 21:04:48 15962:15962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5762256Z STAGE:2023-01-11 21:04:48 15960:15960 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5762528Z STAGE:2023-01-11 21:04:48 15961:15961 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5762764Z STAGE:2023-01-11 21:04:48 15962:15962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5763023Z STAGE:2023-01-11 21:04:48 15960:15960 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5763279Z STAGE:2023-01-11 21:04:48 15961:15961 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5763533Z STAGE:2023-01-11 21:04:48 15962:15962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5764099Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5764254Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5764809Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5764960Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5765521Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5765667Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5765738Z ok (2.178s)
2023-01-11T21:18:21.5765745Z 
2023-01-11T21:18:21.5765932Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5766010Z Ran 1 test in 2.179s
2023-01-11T21:18:21.5766016Z 
2023-01-11T21:18:21.5766077Z OK
2023-01-11T21:18:21.5766083Z 
2023-01-11T21:18:21.5766168Z Generating XML reports...
2023-01-11T21:18:21.5766510Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210446.xml
2023-01-11T21:18:21.5766742Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5767014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5767138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5767405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5767542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5767547Z 
2023-01-11T21:18:21.5767621Z Running tests...
2023-01-11T21:18:21.5767818Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5768050Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16016
2023-01-11T21:18:21.5768208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16017
2023-01-11T21:18:21.5768363Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16018
2023-01-11T21:18:21.5768635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5768760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5769028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5769193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5769398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5769673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5769798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5770078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5770213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5770389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5770647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5770771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5771055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5771189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5771365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5771663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5771961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5772255Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5772422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5772572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5772734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5772982Z STAGE:2023-01-11 21:04:52 16017:16017 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5773223Z STAGE:2023-01-11 21:04:52 16016:16016 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5773459Z STAGE:2023-01-11 21:04:52 16018:16018 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5773705Z STAGE:2023-01-11 21:04:52 16016:16016 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5773949Z STAGE:2023-01-11 21:04:52 16017:16017 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5774195Z STAGE:2023-01-11 21:04:52 16018:16018 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5774456Z STAGE:2023-01-11 21:04:52 16016:16016 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5774701Z STAGE:2023-01-11 21:04:52 16017:16017 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5774950Z STAGE:2023-01-11 21:04:52 16018:16018 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5775191Z STAGE:2023-01-11 21:04:52 16018:16018 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5775430Z STAGE:2023-01-11 21:04:52 16016:16016 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5775665Z STAGE:2023-01-11 21:04:52 16017:16017 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5775908Z STAGE:2023-01-11 21:04:52 16016:16016 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5776177Z STAGE:2023-01-11 21:04:52 16017:16017 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5776447Z STAGE:2023-01-11 21:04:52 16018:16018 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5776699Z STAGE:2023-01-11 21:04:52 16016:16016 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5776941Z STAGE:2023-01-11 21:04:52 16017:16017 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5777190Z STAGE:2023-01-11 21:04:52 16018:16018 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5777430Z STAGE:2023-01-11 21:04:52 16016:16016 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5777667Z STAGE:2023-01-11 21:04:52 16017:16017 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5777905Z STAGE:2023-01-11 21:04:52 16018:16018 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5778486Z STAGE:2023-01-11 21:04:52 16017:16017 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:04:52 16016:16016 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:04:52 16018:16018 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5778493Z 
2023-01-11T21:18:21.5778497Z 
2023-01-11T21:18:21.5778751Z STAGE:2023-01-11 21:04:52 16017:16017 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5779182Z STAGE:2023-01-11 21:04:52 16016:16016 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:04:52 16018:16018 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5779189Z 
2023-01-11T21:18:21.5779255Z ok (2.278s)
2023-01-11T21:18:21.5779260Z 
2023-01-11T21:18:21.5779458Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5779532Z Ran 1 test in 2.279s
2023-01-11T21:18:21.5779540Z 
2023-01-11T21:18:21.5779587Z OK
2023-01-11T21:18:21.5779607Z 
2023-01-11T21:18:21.5779678Z Generating XML reports...
2023-01-11T21:18:21.5780023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210450.xml
2023-01-11T21:18:21.5780253Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5780531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5780654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5780936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5781072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5781078Z 
2023-01-11T21:18:21.5781149Z Running tests...
2023-01-11T21:18:21.5781339Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5781529Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s)
2023-01-11T21:18:21.5781537Z 
2023-01-11T21:18:21.5781829Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5781908Z Ran 1 test in 0.002s
2023-01-11T21:18:21.5781914Z 
2023-01-11T21:18:21.5781988Z OK (skipped=1)
2023-01-11T21:18:21.5781994Z 
2023-01-11T21:18:21.5782078Z Generating XML reports...
2023-01-11T21:18:21.5782421Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210454.xml
2023-01-11T21:18:21.5782655Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5782937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5783048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5783380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5783548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5783555Z 
2023-01-11T21:18:21.5783627Z Running tests...
2023-01-11T21:18:21.5783826Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5784027Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s)
2023-01-11T21:18:21.5784032Z 
2023-01-11T21:18:21.5784224Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5784300Z Ran 1 test in 0.002s
2023-01-11T21:18:21.5784305Z 
2023-01-11T21:18:21.5784364Z OK (skipped=1)
2023-01-11T21:18:21.5784383Z 
2023-01-11T21:18:21.5784454Z Generating XML reports...
2023-01-11T21:18:21.5784794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210456.xml
2023-01-11T21:18:21.5785028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5785306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5785431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5785711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5785847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5785852Z 
2023-01-11T21:18:21.5785926Z Running tests...
2023-01-11T21:18:21.5786109Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5786345Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16092
2023-01-11T21:18:21.5786508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16093
2023-01-11T21:18:21.5786666Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16094
2023-01-11T21:18:21.5786941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5787064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5787344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5787479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5787658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5787919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5788046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5788329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5788466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5788645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5788916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5789040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5789323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5789444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5789622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5789963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5790291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5790591Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5790757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5790923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5791083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5791258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.5791419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.5791600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.5791898Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5792195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5792606Z STAGE:2023-01-11 21:04:59 16094:16094 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:04:59 16093:16093 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5792612Z 
2023-01-11T21:18:21.5792908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5793154Z STAGE:2023-01-11 21:04:59 16092:16092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5793407Z STAGE:2023-01-11 21:04:59 16092:16092 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5793668Z STAGE:2023-01-11 21:04:59 16092:16092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5794081Z STAGE:2023-01-11 21:04:59 16094:16094 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:04:59 16093:16093 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5794087Z 
2023-01-11T21:18:21.5794518Z STAGE:2023-01-11 21:04:59 16094:16094 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:04:59 16093:16093 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5794524Z 
2023-01-11T21:18:21.5794767Z STAGE:2023-01-11 21:04:59 16092:16092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5795147Z STAGE:2023-01-11 21:04:59 16094:16094 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:04:59 16093:16093 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5795169Z 
2023-01-11T21:18:21.5795565Z STAGE:2023-01-11 21:04:59 16092:16092 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:04:59 16094:16094 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5795585Z 
2023-01-11T21:18:21.5795817Z STAGE:2023-01-11 21:04:59 16093:16093 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5796075Z STAGE:2023-01-11 21:04:59 16092:16092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5796334Z STAGE:2023-01-11 21:04:59 16094:16094 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5796585Z STAGE:2023-01-11 21:04:59 16093:16093 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5796872Z STAGE:2023-01-11 21:04:59 16094:16094 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5797141Z STAGE:2023-01-11 21:04:59 16092:16092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5797379Z STAGE:2023-01-11 21:04:59 16093:16093 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5797625Z STAGE:2023-01-11 21:04:59 16092:16092 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5797870Z STAGE:2023-01-11 21:04:59 16093:16093 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5798113Z STAGE:2023-01-11 21:04:59 16092:16092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5798359Z STAGE:2023-01-11 21:04:59 16094:16094 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5798615Z STAGE:2023-01-11 21:04:59 16093:16093 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5798872Z STAGE:2023-01-11 21:04:59 16094:16094 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5798941Z ok (2.274s)
2023-01-11T21:18:21.5798947Z 
2023-01-11T21:18:21.5799150Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5799225Z Ran 1 test in 2.274s
2023-01-11T21:18:21.5799230Z 
2023-01-11T21:18:21.5799291Z OK
2023-01-11T21:18:21.5799296Z 
2023-01-11T21:18:21.5799368Z Generating XML reports...
2023-01-11T21:18:21.5799714Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210457.xml
2023-01-11T21:18:21.5799949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5800224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5800349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5800634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5800772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5800778Z 
2023-01-11T21:18:21.5800851Z Running tests...
2023-01-11T21:18:21.5801053Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5801272Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16157
2023-01-11T21:18:21.5801429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16158
2023-01-11T21:18:21.5801654Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16159
2023-01-11T21:18:21.5801932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5802056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5802340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5802479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5802657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5802916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5803040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5803324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5803460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5803637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5803943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5804096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5804381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5804515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5804680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5804978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5805274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5805567Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5805735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5805902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5806065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5806237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.5806412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.5806572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.5806862Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5807154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5807406Z STAGE:2023-01-11 21:05:03 16159:16159 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5807655Z STAGE:2023-01-11 21:05:03 16158:16158 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5807901Z STAGE:2023-01-11 21:05:03 16158:16158 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5808141Z STAGE:2023-01-11 21:05:03 16159:16159 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5808400Z STAGE:2023-01-11 21:05:03 16158:16158 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5808654Z STAGE:2023-01-11 21:05:03 16159:16159 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5808883Z STAGE:2023-01-11 21:05:03 16158:16158 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5809120Z STAGE:2023-01-11 21:05:03 16159:16159 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5809534Z STAGE:2023-01-11 21:05:03 16159:16159 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:03 16158:16158 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5809540Z 
2023-01-11T21:18:21.5809796Z STAGE:2023-01-11 21:05:03 16158:16158 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5810051Z STAGE:2023-01-11 21:05:03 16159:16159 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5810349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5810420Z ok (2.285s)
2023-01-11T21:18:21.5810425Z 
2023-01-11T21:18:21.5810622Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5810698Z Ran 1 test in 2.286s
2023-01-11T21:18:21.5810736Z 
2023-01-11T21:18:21.5810795Z OK
2023-01-11T21:18:21.5810801Z 
2023-01-11T21:18:21.5810871Z Generating XML reports...
2023-01-11T21:18:21.5811237Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210502.xml
2023-01-11T21:18:21.5811470Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5811747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5811870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5812154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5812289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5812294Z 
2023-01-11T21:18:21.5812366Z Running tests...
2023-01-11T21:18:21.5812552Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5812771Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2023-01-11T21:18:21.5812779Z 
2023-01-11T21:18:21.5812973Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5813048Z Ran 1 test in 0.002s
2023-01-11T21:18:21.5813053Z 
2023-01-11T21:18:21.5813124Z OK (skipped=1)
2023-01-11T21:18:21.5813129Z 
2023-01-11T21:18:21.5813214Z Generating XML reports...
2023-01-11T21:18:21.5813554Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210506.xml
2023-01-11T21:18:21.5813784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5814058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5814168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5814452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5814587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5814592Z 
2023-01-11T21:18:21.5814665Z Running tests...
2023-01-11T21:18:21.5814861Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5815079Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2023-01-11T21:18:21.5815085Z 
2023-01-11T21:18:21.5815275Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5815351Z Ran 1 test in 0.002s
2023-01-11T21:18:21.5815357Z 
2023-01-11T21:18:21.5815426Z OK (skipped=1)
2023-01-11T21:18:21.5815431Z 
2023-01-11T21:18:21.5815503Z Generating XML reports...
2023-01-11T21:18:21.5815838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210507.xml
2023-01-11T21:18:21.5816075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5816348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5816471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5816752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5816887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5816892Z 
2023-01-11T21:18:21.5816968Z Running tests...
2023-01-11T21:18:21.5817162Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5817358Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2023-01-11T21:18:21.5817394Z 
2023-01-11T21:18:21.5817585Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5817659Z Ran 1 test in 0.002s
2023-01-11T21:18:21.5817691Z 
2023-01-11T21:18:21.5817763Z OK (skipped=1)
2023-01-11T21:18:21.5817768Z 
2023-01-11T21:18:21.5817851Z Generating XML reports...
2023-01-11T21:18:21.5818189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210509.xml
2023-01-11T21:18:21.5818419Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5818691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5818801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5819078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5819213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5819219Z 
2023-01-11T21:18:21.5819291Z Running tests...
2023-01-11T21:18:21.5819488Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5819703Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2023-01-11T21:18:21.5819709Z 
2023-01-11T21:18:21.5819903Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5819978Z Ran 1 test in 0.002s
2023-01-11T21:18:21.5819983Z 
2023-01-11T21:18:21.5820053Z OK (skipped=1)
2023-01-11T21:18:21.5820059Z 
2023-01-11T21:18:21.5820129Z Generating XML reports...
2023-01-11T21:18:21.5820462Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210511.xml
2023-01-11T21:18:21.5820692Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5820969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5821096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5821374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5821509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5821514Z 
2023-01-11T21:18:21.5821586Z Running tests...
2023-01-11T21:18:21.5821908Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5822139Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16258
2023-01-11T21:18:21.5822297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16259
2023-01-11T21:18:21.5822457Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16260
2023-01-11T21:18:21.5822733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5822857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5823137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5823272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5823449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5823724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5823833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5824115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5824300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5824509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5824785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5824908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5825188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5825324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5825487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5825784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5826087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5826387Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5826554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5826720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5826882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5826952Z ok (2.179s)
2023-01-11T21:18:21.5826958Z 
2023-01-11T21:18:21.5827157Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5827220Z Ran 1 test in 2.180s
2023-01-11T21:18:21.5827225Z 
2023-01-11T21:18:21.5827285Z OK
2023-01-11T21:18:21.5827290Z 
2023-01-11T21:18:21.5827376Z Generating XML reports...
2023-01-11T21:18:21.5827715Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210512.xml
2023-01-11T21:18:21.5827948Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5828222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5828344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5828625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5828759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5828765Z 
2023-01-11T21:18:21.5828824Z Running tests...
2023-01-11T21:18:21.5829022Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5829263Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16311
2023-01-11T21:18:21.5829426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16312
2023-01-11T21:18:21.5829580Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16313
2023-01-11T21:18:21.5829854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5829979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5830257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5830377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5830553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5830822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5830989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5831298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5831432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5831608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5831877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5831999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5832264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5832397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5832573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5832871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5833169Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5833461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5833627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5833789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5833951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5834112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.5834288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.5834461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.5834756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5835047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5835334Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5835504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:18:21.5835674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:18:21.5835846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T21:18:21.5836127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2023-01-11T21:18:21.5836416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2023-01-11T21:18:21.5836708Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2023-01-11T21:18:21.5836877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:18:21.5837050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T21:18:21.5837219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:18:21.5837546Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2023-01-11T21:18:21.5837862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2023-01-11T21:18:21.5838162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2023-01-11T21:18:21.5838217Z ok (2.484s)
2023-01-11T21:18:21.5838234Z 
2023-01-11T21:18:21.5838421Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5838498Z Ran 1 test in 2.484s
2023-01-11T21:18:21.5838504Z 
2023-01-11T21:18:21.5838564Z OK
2023-01-11T21:18:21.5838569Z 
2023-01-11T21:18:21.5838657Z Generating XML reports...
2023-01-11T21:18:21.5838997Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210516.xml
2023-01-11T21:18:21.5839228Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5839506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5839632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5839901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5840035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5840040Z 
2023-01-11T21:18:21.5840113Z Running tests...
2023-01-11T21:18:21.5840312Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5840501Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2023-01-11T21:18:21.5840507Z 
2023-01-11T21:18:21.5840702Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5840781Z Ran 1 test in 0.002s
2023-01-11T21:18:21.5840787Z 
2023-01-11T21:18:21.5840857Z OK (skipped=1)
2023-01-11T21:18:21.5840862Z 
2023-01-11T21:18:21.5840944Z Generating XML reports...
2023-01-11T21:18:21.5841274Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210521.xml
2023-01-11T21:18:21.5841505Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5841856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5841984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5842269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5842404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5842409Z 
2023-01-11T21:18:21.5842484Z Running tests...
2023-01-11T21:18:21.5842687Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5842930Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16410
2023-01-11T21:18:21.5843093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16411
2023-01-11T21:18:21.5843248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16412
2023-01-11T21:18:21.5843521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5843646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5843958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5844138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5844446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5844911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5845405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5846084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5846540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5847109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5847780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5848235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5848862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5849377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5849880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5850615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5851451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5852311Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5876320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5876947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5877504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5878096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.5878620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.5879505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5880209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.5881102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5882145Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5882849Z STAGE:2023-01-11 21:05:24 16410:16410 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5883538Z STAGE:2023-01-11 21:05:24 16411:16411 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5884291Z STAGE:2023-01-11 21:05:24 16412:16412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5885009Z STAGE:2023-01-11 21:05:24 16410:16410 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5886101Z STAGE:2023-01-11 21:05:24 16411:16411 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:24 16412:16412 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5887160Z STAGE:2023-01-11 21:05:24 16410:16410 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5887524Z 
2023-01-11T21:18:21.5888333Z STAGE:2023-01-11 21:05:24 16412:16412 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:24 16411:16411 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5889021Z 
2023-01-11T21:18:21.5889832Z STAGE:2023-01-11 21:05:24 16410:16410 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:05:24 16412:16412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5890339Z 
2023-01-11T21:18:21.5890787Z STAGE:2023-01-11 21:05:24 16411:16411 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5891578Z STAGE:2023-01-11 21:05:24 16412:16412 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5892371Z STAGE:2023-01-11 21:05:24 16410:16410 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5893159Z STAGE:2023-01-11 21:05:24 16411:16411 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5893960Z STAGE:2023-01-11 21:05:24 16412:16412 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5894789Z STAGE:2023-01-11 21:05:24 16410:16410 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5895619Z STAGE:2023-01-11 21:05:24 16411:16411 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5897015Z STAGE:2023-01-11 21:05:24 16411:16411 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:05:24 16410:16410 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:05:24 16412:16412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5897691Z 
2023-01-11T21:18:21.5897699Z 
2023-01-11T21:18:21.5898158Z STAGE:2023-01-11 21:05:24 16410:16410 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5898939Z STAGE:2023-01-11 21:05:24 16412:16412 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5899738Z STAGE:2023-01-11 21:05:24 16411:16411 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5900522Z STAGE:2023-01-11 21:05:24 16410:16410 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5901276Z STAGE:2023-01-11 21:05:24 16412:16412 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5902274Z STAGE:2023-01-11 21:05:24 16411:16411 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5903704Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5904711Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5906114Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5907103Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5908477Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5909468Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5909891Z ok (2.274s)
2023-01-11T21:18:21.5910070Z 
2023-01-11T21:18:21.5910434Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5910849Z Ran 1 test in 2.274s
2023-01-11T21:18:21.5911048Z 
2023-01-11T21:18:21.5911150Z OK
2023-01-11T21:18:21.5911311Z 
2023-01-11T21:18:21.5911454Z Generating XML reports...
2023-01-11T21:18:21.5912402Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210522.xml
2023-01-11T21:18:21.5913371Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5914230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5914831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5915628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5916240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5916523Z 
2023-01-11T21:18:21.5916655Z Running tests...
2023-01-11T21:18:21.5917218Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5917966Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16475
2023-01-11T21:18:21.5918741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16476
2023-01-11T21:18:21.5919359Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16477
2023-01-11T21:18:21.5920206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5920801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5921688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5922326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5922931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5923804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5924414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5925213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5925830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5926440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5927357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5928230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5928814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5929607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5930156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5930643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5931347Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5932170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5932880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5933396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5933969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5934513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.5935403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5936138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.5936761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.5937267Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5937784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5938262Z STAGE:2023-01-11 21:05:28 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5938679Z STAGE:2023-01-11 21:05:28 16477:16477 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5939105Z STAGE:2023-01-11 21:05:28 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5939544Z STAGE:2023-01-11 21:05:28 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5939982Z STAGE:2023-01-11 21:05:28 16477:16477 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5940396Z STAGE:2023-01-11 21:05:28 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5940843Z STAGE:2023-01-11 21:05:28 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5941295Z STAGE:2023-01-11 21:05:28 16477:16477 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5941855Z STAGE:2023-01-11 21:05:28 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5942278Z STAGE:2023-01-11 21:05:28 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5942709Z STAGE:2023-01-11 21:05:28 16477:16477 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5943133Z STAGE:2023-01-11 21:05:28 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5943881Z STAGE:2023-01-11 21:05:28 16477:16477 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:28 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:28 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5944265Z 
2023-01-11T21:18:21.5944284Z 
2023-01-11T21:18:21.5944878Z STAGE:2023-01-11 21:05:28 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:28 16477:16477 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:28 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5945276Z 
2023-01-11T21:18:21.5945294Z 
2023-01-11T21:18:21.5945524Z STAGE:2023-01-11 21:05:28 16477:16477 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5945949Z STAGE:2023-01-11 21:05:28 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5946372Z STAGE:2023-01-11 21:05:28 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5946800Z STAGE:2023-01-11 21:05:28 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5947381Z STAGE:2023-01-11 21:05:28 16477:16477 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:28 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5947664Z 
2023-01-11T21:18:21.5947923Z STAGE:2023-01-11 21:05:28 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5948549Z STAGE:2023-01-11 21:05:28 16477:16477 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:28 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5948928Z 
2023-01-11T21:18:21.5949547Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5950089Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5950796Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5951338Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5952069Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5952600Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5952827Z ok (2.280s)
2023-01-11T21:18:21.5952930Z 
2023-01-11T21:18:21.5953133Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5953376Z Ran 1 test in 2.280s
2023-01-11T21:18:21.5953489Z 
2023-01-11T21:18:21.5953551Z OK
2023-01-11T21:18:21.5953629Z 
2023-01-11T21:18:21.5953713Z Generating XML reports...
2023-01-11T21:18:21.5954178Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210527.xml
2023-01-11T21:18:21.5954675Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5955128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5955467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5955896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5956241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5956395Z 
2023-01-11T21:18:21.5956468Z Running tests...
2023-01-11T21:18:21.5956767Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5957186Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16540
2023-01-11T21:18:21.5957594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16541
2023-01-11T21:18:21.5957927Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16542
2023-01-11T21:18:21.5958380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5958711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5959127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5959474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5959809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5960272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5960586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5961012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5961399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5961819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5962296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5962626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5963055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5963386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5963721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5964222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5964751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5965256Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5965643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5965990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5966320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5967135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.5967724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.5968319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.5968813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5969330Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5969844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.5970309Z STAGE:2023-01-11 21:05:32 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5970890Z STAGE:2023-01-11 21:05:32 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:05:32 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5971169Z 
2023-01-11T21:18:21.5971419Z STAGE:2023-01-11 21:05:32 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5972023Z STAGE:2023-01-11 21:05:32 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:32 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5972309Z 
2023-01-11T21:18:21.5972570Z STAGE:2023-01-11 21:05:32 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5973194Z STAGE:2023-01-11 21:05:32 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:32 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5973476Z 
2023-01-11T21:18:21.5973723Z STAGE:2023-01-11 21:05:32 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5974311Z STAGE:2023-01-11 21:05:32 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:05:32 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5974639Z 
2023-01-11T21:18:21.5974918Z STAGE:2023-01-11 21:05:32 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5975521Z STAGE:2023-01-11 21:05:32 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:32 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5975805Z 
2023-01-11T21:18:21.5976052Z STAGE:2023-01-11 21:05:32 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5976672Z STAGE:2023-01-11 21:05:32 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:32 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5976969Z 
2023-01-11T21:18:21.5977212Z STAGE:2023-01-11 21:05:32 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5977638Z STAGE:2023-01-11 21:05:32 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5978061Z STAGE:2023-01-11 21:05:32 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.5978605Z STAGE:2023-01-11 21:05:32 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5979038Z STAGE:2023-01-11 21:05:32 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.5979645Z STAGE:2023-01-11 21:05:32 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:32 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5979935Z 
2023-01-11T21:18:21.5980196Z STAGE:2023-01-11 21:05:32 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5980633Z STAGE:2023-01-11 21:05:32 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.5981409Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5982135Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5982877Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5983403Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5984131Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.5984667Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.5984906Z ok (2.279s)
2023-01-11T21:18:21.5985010Z 
2023-01-11T21:18:21.5985198Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5985441Z Ran 1 test in 2.280s
2023-01-11T21:18:21.5985556Z 
2023-01-11T21:18:21.5985617Z OK
2023-01-11T21:18:21.5985707Z 
2023-01-11T21:18:21.5985792Z Generating XML reports...
2023-01-11T21:18:21.5986244Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210531.xml
2023-01-11T21:18:21.5986740Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.5987208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5987613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5988085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5988437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5988602Z 
2023-01-11T21:18:21.5988675Z Running tests...
2023-01-11T21:18:21.5988970Z ----------------------------------------------------------------------
2023-01-11T21:18:21.5989386Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16605
2023-01-11T21:18:21.5989804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16606
2023-01-11T21:18:21.5990122Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16607
2023-01-11T21:18:21.5991041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5991594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5992035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5992370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5992704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.5993175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5993501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5993917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5994256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5994593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.5995079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5995576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.5995905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.5996332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.5996664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.5996996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.5997495Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5998015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.5998389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.5998738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.5999081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.5999428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.5999789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6000154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6000645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6001227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6001813Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6002451Z STAGE:2023-01-11 21:05:37 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:05:37 16605:16605 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6002967Z STAGE:2023-01-11 21:05:37 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6003162Z 
2023-01-11T21:18:21.6003560Z STAGE:2023-01-11 21:05:37 16605:16605 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:37 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6004094Z STAGE:2023-01-11 21:05:37 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6004285Z 
2023-01-11T21:18:21.6004899Z STAGE:2023-01-11 21:05:37 16605:16605 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:37 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:37 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6005299Z 
2023-01-11T21:18:21.6005304Z 
2023-01-11T21:18:21.6005706Z STAGE:2023-01-11 21:05:37 16605:16605 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:05:37 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6006216Z STAGE:2023-01-11 21:05:37 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6006404Z 
2023-01-11T21:18:21.6006654Z STAGE:2023-01-11 21:05:37 16605:16605 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6007089Z STAGE:2023-01-11 21:05:37 16605:16605 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6007532Z STAGE:2023-01-11 21:05:37 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6007964Z STAGE:2023-01-11 21:05:37 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6008573Z STAGE:2023-01-11 21:05:37 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:37 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6008858Z 
2023-01-11T21:18:21.6009102Z STAGE:2023-01-11 21:05:37 16605:16605 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6009803Z STAGE:2023-01-11 21:05:37 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:05:37 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6010079Z 
2023-01-11T21:18:21.6010331Z STAGE:2023-01-11 21:05:37 16605:16605 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6010763Z STAGE:2023-01-11 21:05:37 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6011193Z STAGE:2023-01-11 21:05:37 16605:16605 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6011631Z STAGE:2023-01-11 21:05:37 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6012074Z STAGE:2023-01-11 21:05:37 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6012522Z STAGE:2023-01-11 21:05:37 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6013271Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6013892Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6014628Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6015171Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6015909Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6016435Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6016670Z ok (2.276s)
2023-01-11T21:18:21.6016771Z 
2023-01-11T21:18:21.6016975Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6017205Z Ran 1 test in 2.276s
2023-01-11T21:18:21.6017316Z 
2023-01-11T21:18:21.6017377Z OK
2023-01-11T21:18:21.6017468Z 
2023-01-11T21:18:21.6017551Z Generating XML reports...
2023-01-11T21:18:21.6018013Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210535.xml
2023-01-11T21:18:21.6018495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6018958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6019296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6019715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6020062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6020223Z 
2023-01-11T21:18:21.6020297Z Running tests...
2023-01-11T21:18:21.6020599Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6020997Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16670
2023-01-11T21:18:21.6021408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16671
2023-01-11T21:18:21.6021843Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16672
2023-01-11T21:18:21.6022298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6022620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6023052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6023401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6023726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6024196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6024529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6024958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6025286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6025622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6026086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6026467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6026936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6027291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6027628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6028111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6028632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6029153Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6029541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6029881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6030230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6030588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6030946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6031310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6031802Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6032317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6032936Z STAGE:2023-01-11 21:05:41 16672:16672 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:05:41 16671:16671 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6033218Z 
2023-01-11T21:18:21.6033628Z STAGE:2023-01-11 21:05:41 16672:16672 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:41 16671:16671 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6033915Z 
2023-01-11T21:18:21.6034348Z STAGE:2023-01-11 21:05:41 16672:16672 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:41 16671:16671 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6034643Z 
2023-01-11T21:18:21.6035043Z STAGE:2023-01-11 21:05:41 16672:16672 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:05:41 16671:16671 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6035326Z 
2023-01-11T21:18:21.6035739Z STAGE:2023-01-11 21:05:41 16672:16672 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:41 16671:16671 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6036026Z 
2023-01-11T21:18:21.6036455Z STAGE:2023-01-11 21:05:41 16672:16672 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:41 16671:16671 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6036753Z 
2023-01-11T21:18:21.6037051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6037831Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6038416Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6039176Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6039716Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6039943Z ok (2.287s)
2023-01-11T21:18:21.6040045Z 
2023-01-11T21:18:21.6040246Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6040488Z Ran 1 test in 2.287s
2023-01-11T21:18:21.6040601Z 
2023-01-11T21:18:21.6040662Z OK
2023-01-11T21:18:21.6040740Z 
2023-01-11T21:18:21.6040823Z Generating XML reports...
2023-01-11T21:18:21.6041364Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210539.xml
2023-01-11T21:18:21.6041947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6042403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6042735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6043167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6043511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6043661Z 
2023-01-11T21:18:21.6043734Z Running tests...
2023-01-11T21:18:21.6044040Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6044456Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16731
2023-01-11T21:18:21.6045028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16732
2023-01-11T21:18:21.6045362Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16733
2023-01-11T21:18:21.6045816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6046150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6046569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6046913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6047249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6047709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6048036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6048467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6048817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6049140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6049614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6049945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6050369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6050697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6051028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6051579Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6052121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6052641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6053028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6053380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6053711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6054068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6054438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6054808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6055291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6055809Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6056282Z STAGE:2023-01-11 21:05:45 16732:16732 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6056717Z STAGE:2023-01-11 21:05:45 16733:16733 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6057304Z STAGE:2023-01-11 21:05:45 16733:16733 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:45 16732:16732 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6057604Z 
2023-01-11T21:18:21.6058200Z STAGE:2023-01-11 21:05:45 16732:16732 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:45 16733:16733 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6058590Z 
2023-01-11T21:18:21.6059065Z STAGE:2023-01-11 21:05:45 16733:16733 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6059611Z STAGE:2023-01-11 21:05:45 16732:16732 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6060198Z STAGE:2023-01-11 21:05:45 16733:16733 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:45 16732:16732 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6060481Z 
2023-01-11T21:18:21.6060920Z STAGE:2023-01-11 21:05:45 16733:16733 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:45 16732:16732 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6061220Z 
2023-01-11T21:18:21.6061522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6062440Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6062987Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6063713Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6064256Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6064586Z ok (2.277s)
2023-01-11T21:18:21.6064690Z 
2023-01-11T21:18:21.6064898Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6065165Z Ran 1 test in 2.277s
2023-01-11T21:18:21.6065281Z 
2023-01-11T21:18:21.6065346Z OK
2023-01-11T21:18:21.6065442Z 
2023-01-11T21:18:21.6065529Z Generating XML reports...
2023-01-11T21:18:21.6065998Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210543.xml
2023-01-11T21:18:21.6066485Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6066957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6067291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6067713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6068064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6068231Z 
2023-01-11T21:18:21.6068303Z Running tests...
2023-01-11T21:18:21.6068614Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6069019Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16792
2023-01-11T21:18:21.6069441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16793
2023-01-11T21:18:21.6069778Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16794
2023-01-11T21:18:21.6070233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6070549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6070983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6071332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6071657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6072131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6072462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6072885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6073220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6073557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6074031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6074363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6074782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6075133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6075472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6075961Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6076494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6077017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6077404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6077800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6078180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6078543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6078910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6079269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6079759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6080279Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6080752Z STAGE:2023-01-11 21:05:49 16793:16793 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6081174Z STAGE:2023-01-11 21:05:49 16794:16794 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6081863Z STAGE:2023-01-11 21:05:49 16793:16793 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:49 16794:16794 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6082154Z 
2023-01-11T21:18:21.6082422Z STAGE:2023-01-11 21:05:49 16794:16794 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6082876Z STAGE:2023-01-11 21:05:49 16793:16793 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6083298Z STAGE:2023-01-11 21:05:49 16794:16794 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6083723Z STAGE:2023-01-11 21:05:49 16793:16793 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6084157Z STAGE:2023-01-11 21:05:49 16794:16794 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6084597Z STAGE:2023-01-11 21:05:49 16793:16793 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6085032Z STAGE:2023-01-11 21:05:49 16794:16794 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6085484Z STAGE:2023-01-11 21:05:49 16793:16793 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6085981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6086779Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6087319Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6088056Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6088597Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6088841Z ok (2.380s)
2023-01-11T21:18:21.6088946Z 
2023-01-11T21:18:21.6089135Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6089381Z Ran 1 test in 2.380s
2023-01-11T21:18:21.6089494Z 
2023-01-11T21:18:21.6089556Z OK
2023-01-11T21:18:21.6089647Z 
2023-01-11T21:18:21.6089731Z Generating XML reports...
2023-01-11T21:18:21.6090182Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210547.xml
2023-01-11T21:18:21.6090728Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6091232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6091556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6091991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6092339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6092503Z 
2023-01-11T21:18:21.6092576Z Running tests...
2023-01-11T21:18:21.6092868Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6093285Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16853
2023-01-11T21:18:21.6093693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16854
2023-01-11T21:18:21.6094017Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16855
2023-01-11T21:18:21.6094491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6094824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6095256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6095592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6095929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6096407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6096741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6097159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6097509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6097843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6098303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6098632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6099066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6099414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6099736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6100237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6100762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6101285Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6101819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6102177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6102533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6102896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6103255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6103691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6104223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6104738Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6105371Z STAGE:2023-01-11 21:05:53 16855:16855 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:05:53 16854:16854 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6105654Z 
2023-01-11T21:18:21.6106069Z STAGE:2023-01-11 21:05:53 16855:16855 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:53 16854:16854 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6106360Z 
2023-01-11T21:18:21.6106792Z STAGE:2023-01-11 21:05:53 16855:16855 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:53 16854:16854 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6107093Z 
2023-01-11T21:18:21.6107491Z STAGE:2023-01-11 21:05:53 16855:16855 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:05:53 16854:16854 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6107764Z 
2023-01-11T21:18:21.6108179Z STAGE:2023-01-11 21:05:53 16855:16855 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:53 16854:16854 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6108461Z 
2023-01-11T21:18:21.6108707Z STAGE:2023-01-11 21:05:53 16855:16855 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6109163Z STAGE:2023-01-11 21:05:53 16854:16854 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6109658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6110455Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6111010Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6111735Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6112273Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6112518Z ok (2.279s)
2023-01-11T21:18:21.6112623Z 
2023-01-11T21:18:21.6112827Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6113055Z Ran 1 test in 2.279s
2023-01-11T21:18:21.6113169Z 
2023-01-11T21:18:21.6113234Z OK
2023-01-11T21:18:21.6113329Z 
2023-01-11T21:18:21.6113415Z Generating XML reports...
2023-01-11T21:18:21.6113866Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210551.xml
2023-01-11T21:18:21.6114371Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6114836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6115169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6115588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6115940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6116140Z 
2023-01-11T21:18:21.6116216Z Running tests...
2023-01-11T21:18:21.6116555Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6116957Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16914
2023-01-11T21:18:21.6117365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16915
2023-01-11T21:18:21.6117703Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16916
2023-01-11T21:18:21.6118144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6118477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6118908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6119257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6119581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6120058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6120387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6120819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6120941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6121120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6121400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6121525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6121904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6122043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6122222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6122527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6122811Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6123115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6123282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6123536Z STAGE:2023-01-11 21:05:57 16914:16914 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6123709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6123871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6124122Z STAGE:2023-01-11 21:05:57 16915:16915 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6124366Z STAGE:2023-01-11 21:05:57 16916:16916 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6124621Z STAGE:2023-01-11 21:05:57 16916:16916 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6124850Z STAGE:2023-01-11 21:05:57 16915:16915 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6125097Z STAGE:2023-01-11 21:05:57 16914:16914 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6125576Z STAGE:2023-01-11 21:05:57 16915:16915 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:57 16916:16916 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6125610Z 
2023-01-11T21:18:21.6125874Z STAGE:2023-01-11 21:05:57 16914:16914 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6126118Z STAGE:2023-01-11 21:05:57 16915:16915 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6126364Z STAGE:2023-01-11 21:05:57 16916:16916 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6126604Z STAGE:2023-01-11 21:05:57 16914:16914 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6127020Z STAGE:2023-01-11 21:05:57 16916:16916 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:57 16915:16915 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6127030Z 
2023-01-11T21:18:21.6127279Z STAGE:2023-01-11 21:05:57 16914:16914 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6127542Z STAGE:2023-01-11 21:05:57 16916:16916 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6127800Z STAGE:2023-01-11 21:05:57 16915:16915 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6128040Z STAGE:2023-01-11 21:05:57 16914:16914 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6128285Z STAGE:2023-01-11 21:05:57 16916:16916 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6128524Z STAGE:2023-01-11 21:05:57 16915:16915 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6128765Z STAGE:2023-01-11 21:05:57 16914:16914 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6129178Z STAGE:2023-01-11 21:05:57 16916:16916 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:05:57 16915:16915 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6129187Z 
2023-01-11T21:18:21.6129437Z STAGE:2023-01-11 21:05:57 16914:16914 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6129868Z STAGE:2023-01-11 21:05:57 16916:16916 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:05:57 16915:16915 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6129874Z 
2023-01-11T21:18:21.6130128Z STAGE:2023-01-11 21:05:57 16914:16914 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6130699Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6130858Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6131415Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6131561Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6132108Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6132258Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6132315Z ok (2.382s)
2023-01-11T21:18:21.6132352Z 
2023-01-11T21:18:21.6132562Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6132640Z Ran 1 test in 2.382s
2023-01-11T21:18:21.6132646Z 
2023-01-11T21:18:21.6132705Z OK
2023-01-11T21:18:21.6132739Z 
2023-01-11T21:18:21.6132826Z Generating XML reports...
2023-01-11T21:18:21.6133175Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210555.xml
2023-01-11T21:18:21.6133411Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6133687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6133799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6134082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6134219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6134227Z 
2023-01-11T21:18:21.6134301Z Running tests...
2023-01-11T21:18:21.6134498Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6134766Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16970
2023-01-11T21:18:21.6134926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16971
2023-01-11T21:18:21.6135084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16972
2023-01-11T21:18:21.6135360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6135470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6135755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6135895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6136072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6136356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6136479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6136762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6136896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6137059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6137332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6137458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6137746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6137885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6138064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6138369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6138668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6138969Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6139134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6139316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6139482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6140083Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6140239Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6140802Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6140950Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6141518Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6141792Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6141864Z ok (2.175s)
2023-01-11T21:18:21.6141870Z 
2023-01-11T21:18:21.6142076Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6142138Z Ran 1 test in 2.175s
2023-01-11T21:18:21.6142158Z 
2023-01-11T21:18:21.6142207Z OK
2023-01-11T21:18:21.6142211Z 
2023-01-11T21:18:21.6142296Z Generating XML reports...
2023-01-11T21:18:21.6142639Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210600.xml
2023-01-11T21:18:21.6142880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6143161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6143290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6143579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6143721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6143727Z 
2023-01-11T21:18:21.6143788Z Running tests...
2023-01-11T21:18:21.6143984Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6144225Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17023
2023-01-11T21:18:21.6144383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17024
2023-01-11T21:18:21.6144543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17025
2023-01-11T21:18:21.6144817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6144943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6145223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6145363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6145527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6145829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6146100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6146292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6146654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6146853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6147099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6147644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6147870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6148170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6148308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6148484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6148795Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6149094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6149260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6149423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6149589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6149826Z STAGE:2023-01-11 21:06:05 17023:17023 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6150063Z STAGE:2023-01-11 21:06:05 17025:17025 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6150315Z STAGE:2023-01-11 21:06:05 17024:17024 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6150567Z STAGE:2023-01-11 21:06:05 17023:17023 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6150986Z STAGE:2023-01-11 21:06:05 17024:17024 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:05 17025:17025 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6150993Z 
2023-01-11T21:18:21.6151250Z STAGE:2023-01-11 21:06:05 17023:17023 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6151688Z STAGE:2023-01-11 21:06:05 17024:17024 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:05 17025:17025 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6151694Z 
2023-01-11T21:18:21.6152093Z STAGE:2023-01-11 21:06:05 17023:17023 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:05 17025:17025 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6152101Z 
2023-01-11T21:18:21.6152345Z STAGE:2023-01-11 21:06:05 17024:17024 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6152762Z STAGE:2023-01-11 21:06:05 17024:17024 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:05 17025:17025 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6152768Z 
2023-01-11T21:18:21.6153018Z STAGE:2023-01-11 21:06:05 17023:17023 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6153447Z STAGE:2023-01-11 21:06:05 17024:17024 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:05 17025:17025 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6153453Z 
2023-01-11T21:18:21.6153715Z STAGE:2023-01-11 21:06:05 17023:17023 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6154343Z STAGE:2023-01-11 21:06:05 17025:17025 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:05 17024:17024 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:05 17023:17023 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6154351Z 
2023-01-11T21:18:21.6154355Z 
2023-01-11T21:18:21.6154766Z STAGE:2023-01-11 21:06:05 17023:17023 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:05 17024:17024 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6155015Z STAGE:2023-01-11 21:06:05 17025:17025 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6155020Z 
2023-01-11T21:18:21.6155262Z STAGE:2023-01-11 21:06:05 17023:17023 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6155687Z STAGE:2023-01-11 21:06:05 17024:17024 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:05 17025:17025 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6155696Z 
2023-01-11T21:18:21.6156267Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6156426Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6156981Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6157132Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6157698Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6157845Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6157913Z ok (2.274s)
2023-01-11T21:18:21.6157918Z 
2023-01-11T21:18:21.6158119Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6158196Z Ran 1 test in 2.274s
2023-01-11T21:18:21.6158201Z 
2023-01-11T21:18:21.6158260Z OK
2023-01-11T21:18:21.6158266Z 
2023-01-11T21:18:21.6158337Z Generating XML reports...
2023-01-11T21:18:21.6158686Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210604.xml
2023-01-11T21:18:21.6158924Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6159204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6159330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6159618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6159754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6159760Z 
2023-01-11T21:18:21.6159832Z Running tests...
2023-01-11T21:18:21.6160027Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6160260Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17079
2023-01-11T21:18:21.6160422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17080
2023-01-11T21:18:21.6160609Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17081
2023-01-11T21:18:21.6160912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6161034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6161325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6161462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6161727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6161990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6162115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6162399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6162540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6162722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6163003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6163132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6163421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6163555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6163720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6164021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6164329Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6164631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6164796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6164960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6165122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6165374Z STAGE:2023-01-11 21:06:10 17081:17081 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6165615Z STAGE:2023-01-11 21:06:10 17080:17080 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6165838Z STAGE:2023-01-11 21:06:10 17079:17079 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6166095Z STAGE:2023-01-11 21:06:10 17079:17079 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6166512Z STAGE:2023-01-11 21:06:10 17080:17080 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:10 17081:17081 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6166518Z 
2023-01-11T21:18:21.6166779Z STAGE:2023-01-11 21:06:10 17079:17079 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6167219Z STAGE:2023-01-11 21:06:10 17081:17081 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:10 17080:17080 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6167225Z 
2023-01-11T21:18:21.6167624Z STAGE:2023-01-11 21:06:10 17080:17080 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:10 17081:17081 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6167665Z 
2023-01-11T21:18:21.6167942Z STAGE:2023-01-11 21:06:10 17079:17079 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6168516Z STAGE:2023-01-11 21:06:10 17079:17079 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:10 17081:17081 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:10 17080:17080 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6168522Z 
2023-01-11T21:18:21.6168527Z 
2023-01-11T21:18:21.6169134Z STAGE:2023-01-11 21:06:10 17080:17080 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:10 17079:17079 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:10 17081:17081 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6169140Z 
2023-01-11T21:18:21.6169147Z 
2023-01-11T21:18:21.6169391Z STAGE:2023-01-11 21:06:10 17081:17081 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6169634Z STAGE:2023-01-11 21:06:10 17080:17080 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6169873Z STAGE:2023-01-11 21:06:10 17079:17079 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6170125Z STAGE:2023-01-11 21:06:10 17079:17079 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6170367Z STAGE:2023-01-11 21:06:10 17080:17080 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6170791Z STAGE:2023-01-11 21:06:10 17081:17081 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:10 17079:17079 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6170797Z 
2023-01-11T21:18:21.6171051Z STAGE:2023-01-11 21:06:10 17080:17080 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6171300Z STAGE:2023-01-11 21:06:10 17081:17081 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6171874Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6172031Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6172587Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6172740Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6173308Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6173459Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6173527Z ok (2.275s)
2023-01-11T21:18:21.6173533Z 
2023-01-11T21:18:21.6173738Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6173817Z Ran 1 test in 2.275s
2023-01-11T21:18:21.6173822Z 
2023-01-11T21:18:21.6173884Z OK
2023-01-11T21:18:21.6173889Z 
2023-01-11T21:18:21.6173977Z Generating XML reports...
2023-01-11T21:18:21.6174306Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210608.xml
2023-01-11T21:18:21.6174541Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6174851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6175003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6175292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6175428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6175434Z 
2023-01-11T21:18:21.6175509Z Running tests...
2023-01-11T21:18:21.6175704Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6175933Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17135
2023-01-11T21:18:21.6176092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17136
2023-01-11T21:18:21.6176249Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17137
2023-01-11T21:18:21.6176525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6176651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6176934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6177068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6177244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6177517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6177627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6177910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6178044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6178222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6178495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6178619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6178901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6179035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6179211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6179498Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6179799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6180098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6180262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6180427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6180588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6180840Z STAGE:2023-01-11 21:06:14 17137:17137 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6181084Z STAGE:2023-01-11 21:06:14 17136:17136 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6181323Z STAGE:2023-01-11 21:06:14 17135:17135 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6181636Z STAGE:2023-01-11 21:06:14 17135:17135 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6182042Z STAGE:2023-01-11 21:06:14 17137:17137 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6182286Z STAGE:2023-01-11 21:06:14 17136:17136 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6182546Z STAGE:2023-01-11 21:06:14 17135:17135 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6182805Z STAGE:2023-01-11 21:06:14 17137:17137 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6183061Z STAGE:2023-01-11 21:06:14 17136:17136 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6183465Z STAGE:2023-01-11 21:06:14 17136:17136 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:14 17135:17135 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6183710Z STAGE:2023-01-11 21:06:14 17137:17137 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6183716Z 
2023-01-11T21:18:21.6183962Z STAGE:2023-01-11 21:06:14 17136:17136 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6184204Z STAGE:2023-01-11 21:06:14 17136:17136 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6184614Z STAGE:2023-01-11 21:06:14 17135:17135 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:14 17137:17137 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6184622Z 
2023-01-11T21:18:21.6184880Z STAGE:2023-01-11 21:06:14 17137:17137 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6185133Z STAGE:2023-01-11 21:06:14 17135:17135 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6185381Z STAGE:2023-01-11 21:06:14 17136:17136 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6185782Z STAGE:2023-01-11 21:06:14 17137:17137 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:14 17135:17135 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6185789Z 
2023-01-11T21:18:21.6186037Z STAGE:2023-01-11 21:06:14 17135:17135 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6186280Z STAGE:2023-01-11 21:06:14 17137:17137 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6186520Z STAGE:2023-01-11 21:06:14 17136:17136 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6186776Z STAGE:2023-01-11 21:06:14 17135:17135 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6187031Z STAGE:2023-01-11 21:06:14 17137:17137 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6187273Z STAGE:2023-01-11 21:06:14 17136:17136 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6187845Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6187999Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6188553Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6188702Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6189334Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:18:21.6189480Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:18:21.6189549Z ok (2.273s)
2023-01-11T21:18:21.6189556Z 
2023-01-11T21:18:21.6189755Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6189831Z Ran 1 test in 2.274s
2023-01-11T21:18:21.6189837Z 
2023-01-11T21:18:21.6189897Z OK
2023-01-11T21:18:21.6189902Z 
2023-01-11T21:18:21.6189974Z Generating XML reports...
2023-01-11T21:18:21.6190318Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210612.xml
2023-01-11T21:18:21.6190553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6190832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6190960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6191275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6191477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6191484Z 
2023-01-11T21:18:21.6191604Z Running tests...
2023-01-11T21:18:21.6191969Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6192372Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17191
2023-01-11T21:18:21.6192531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17192
2023-01-11T21:18:21.6192690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17193
2023-01-11T21:18:21.6192978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6193104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6193384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6193520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6193700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6193960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6194084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6194368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6194509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6194690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6194964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6195089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6195369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6195505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6195670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6195974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6196332Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6196663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6196831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6196998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6197162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6197231Z ok (2.273s)
2023-01-11T21:18:21.6197236Z 
2023-01-11T21:18:21.6197439Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6197501Z Ran 1 test in 2.273s
2023-01-11T21:18:21.6197508Z 
2023-01-11T21:18:21.6197571Z OK
2023-01-11T21:18:21.6197576Z 
2023-01-11T21:18:21.6197665Z Generating XML reports...
2023-01-11T21:18:21.6198008Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210616.xml
2023-01-11T21:18:21.6198246Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6198521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6198646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6198932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6199055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6199075Z 
2023-01-11T21:18:21.6199136Z Running tests...
2023-01-11T21:18:21.6199333Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6199575Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17244
2023-01-11T21:18:21.6199738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17245
2023-01-11T21:18:21.6199896Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17246
2023-01-11T21:18:21.6200171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6200294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6200579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6200701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6200880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6201155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6201282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6201566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6201772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6201948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6202515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6202701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6203053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6203190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6203412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6203754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6204052Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6204346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6204514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6204677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6204827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6205000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6205178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6205352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6205646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6205941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6206227Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6206479Z STAGE:2023-01-11 21:06:22 17245:17245 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6206719Z STAGE:2023-01-11 21:06:22 17244:17244 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6206957Z STAGE:2023-01-11 21:06:22 17246:17246 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6207194Z STAGE:2023-01-11 21:06:22 17244:17244 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6207436Z STAGE:2023-01-11 21:06:22 17246:17246 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6207681Z STAGE:2023-01-11 21:06:22 17245:17245 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6207939Z STAGE:2023-01-11 21:06:22 17244:17244 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6208195Z STAGE:2023-01-11 21:06:22 17246:17246 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6208448Z STAGE:2023-01-11 21:06:22 17245:17245 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6209010Z STAGE:2023-01-11 21:06:22 17244:17244 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:22 17246:17246 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:22 17245:17245 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6209019Z 
2023-01-11T21:18:21.6209025Z 
2023-01-11T21:18:21.6209435Z STAGE:2023-01-11 21:06:22 17244:17244 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:22 17246:17246 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6209441Z 
2023-01-11T21:18:21.6209687Z STAGE:2023-01-11 21:06:22 17245:17245 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6209941Z STAGE:2023-01-11 21:06:22 17246:17246 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6210194Z STAGE:2023-01-11 21:06:22 17244:17244 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6210438Z STAGE:2023-01-11 21:06:22 17245:17245 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6210716Z STAGE:2023-01-11 21:06:22 17246:17246 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6210984Z STAGE:2023-01-11 21:06:22 17244:17244 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6211221Z STAGE:2023-01-11 21:06:22 17245:17245 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6211802Z STAGE:2023-01-11 21:06:22 17246:17246 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:22 17244:17244 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:22 17245:17245 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6211808Z 
2023-01-11T21:18:21.6211813Z 
2023-01-11T21:18:21.6212066Z STAGE:2023-01-11 21:06:22 17246:17246 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6212319Z STAGE:2023-01-11 21:06:22 17245:17245 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6212576Z STAGE:2023-01-11 21:06:22 17244:17244 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6212643Z ok (2.380s)
2023-01-11T21:18:21.6212649Z 
2023-01-11T21:18:21.6212848Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6212912Z Ran 1 test in 2.380s
2023-01-11T21:18:21.6212929Z 
2023-01-11T21:18:21.6212977Z OK
2023-01-11T21:18:21.6212982Z 
2023-01-11T21:18:21.6213065Z Generating XML reports...
2023-01-11T21:18:21.6213410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210620.xml
2023-01-11T21:18:21.6213642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6213920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6214047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6214334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6214469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6214475Z 
2023-01-11T21:18:21.6214535Z Running tests...
2023-01-11T21:18:21.6214731Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6214969Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17309
2023-01-11T21:18:21.6215128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17310
2023-01-11T21:18:21.6215282Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17311
2023-01-11T21:18:21.6215554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6215680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6215962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6216095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6216260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6216532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6216657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6216937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6217072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6217290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6217590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6217714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6217980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6218113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6218287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6218586Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6218887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6219182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6219348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6219512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6219674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6219835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6220133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6220307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6220483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6220784Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6221080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6221331Z STAGE:2023-01-11 21:06:26 17309:17309 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6221852Z STAGE:2023-01-11 21:06:26 17311:17311 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:26 17310:17310 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6221859Z 
2023-01-11T21:18:21.6222111Z STAGE:2023-01-11 21:06:26 17309:17309 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6222371Z STAGE:2023-01-11 21:06:26 17309:17309 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6222605Z STAGE:2023-01-11 21:06:26 17311:17311 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6222853Z STAGE:2023-01-11 21:06:26 17310:17310 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6223110Z STAGE:2023-01-11 21:06:26 17311:17311 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6223368Z STAGE:2023-01-11 21:06:26 17310:17310 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6223611Z STAGE:2023-01-11 21:06:26 17309:17309 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6224006Z STAGE:2023-01-11 21:06:26 17311:17311 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:26 17310:17310 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6224012Z 
2023-01-11T21:18:21.6224424Z STAGE:2023-01-11 21:06:26 17311:17311 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:26 17309:17309 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6224488Z 
2023-01-11T21:18:21.6224778Z STAGE:2023-01-11 21:06:26 17310:17310 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6225213Z STAGE:2023-01-11 21:06:26 17309:17309 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:26 17311:17311 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6225218Z 
2023-01-11T21:18:21.6225476Z STAGE:2023-01-11 21:06:26 17310:17310 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6225870Z STAGE:2023-01-11 21:06:26 17309:17309 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:26 17311:17311 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6225876Z 
2023-01-11T21:18:21.6226117Z STAGE:2023-01-11 21:06:26 17310:17310 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6226352Z STAGE:2023-01-11 21:06:26 17311:17311 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6226595Z STAGE:2023-01-11 21:06:26 17309:17309 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6226837Z STAGE:2023-01-11 21:06:26 17310:17310 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6227093Z STAGE:2023-01-11 21:06:26 17311:17311 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6227350Z STAGE:2023-01-11 21:06:26 17309:17309 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6227606Z STAGE:2023-01-11 21:06:26 17310:17310 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6227674Z ok (2.271s)
2023-01-11T21:18:21.6227680Z 
2023-01-11T21:18:21.6227877Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6227954Z Ran 1 test in 2.272s
2023-01-11T21:18:21.6227960Z 
2023-01-11T21:18:21.6228007Z OK
2023-01-11T21:18:21.6228014Z 
2023-01-11T21:18:21.6228098Z Generating XML reports...
2023-01-11T21:18:21.6228443Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210624.xml
2023-01-11T21:18:21.6228675Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6228953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6229076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6229356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6229495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6229502Z 
2023-01-11T21:18:21.6229574Z Running tests...
2023-01-11T21:18:21.6229761Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6230007Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17374
2023-01-11T21:18:21.6230165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17375
2023-01-11T21:18:21.6230318Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17376
2023-01-11T21:18:21.6230592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6230713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6230994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6231127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6231292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6231597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6231749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6232031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6232163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6232342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6232612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6232735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6233014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6233139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6233313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6233617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6233914Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6234210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6234374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6234538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6234698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6234874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6235038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6235208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6235503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6235797Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6236090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6236338Z STAGE:2023-01-11 21:06:30 17375:17375 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6236743Z STAGE:2023-01-11 21:06:30 17376:17376 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:30 17374:17374 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6236749Z 
2023-01-11T21:18:21.6236999Z STAGE:2023-01-11 21:06:30 17374:17374 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6237251Z STAGE:2023-01-11 21:06:30 17376:17376 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6237508Z STAGE:2023-01-11 21:06:30 17374:17374 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6237750Z STAGE:2023-01-11 21:06:30 17376:17376 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6237990Z STAGE:2023-01-11 21:06:30 17376:17376 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6238231Z STAGE:2023-01-11 21:06:30 17374:17374 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6238507Z STAGE:2023-01-11 21:06:30 17375:17375 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6238789Z STAGE:2023-01-11 21:06:30 17375:17375 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6239033Z STAGE:2023-01-11 21:06:30 17375:17375 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6239278Z STAGE:2023-01-11 21:06:30 17376:17376 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6239691Z STAGE:2023-01-11 21:06:30 17375:17375 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:30 17374:17374 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6239697Z 
2023-01-11T21:18:21.6239952Z STAGE:2023-01-11 21:06:30 17376:17376 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6240376Z STAGE:2023-01-11 21:06:30 17374:17374 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:30 17375:17375 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6240385Z 
2023-01-11T21:18:21.6240616Z STAGE:2023-01-11 21:06:30 17376:17376 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6240854Z STAGE:2023-01-11 21:06:30 17375:17375 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6241089Z STAGE:2023-01-11 21:06:30 17374:17374 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6241736Z STAGE:2023-01-11 21:06:30 17375:17375 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:30 17374:17374 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:30 17376:17376 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6241742Z 
2023-01-11T21:18:21.6241746Z 
2023-01-11T21:18:21.6242188Z STAGE:2023-01-11 21:06:30 17375:17375 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:30 17376:17376 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6242196Z 
2023-01-11T21:18:21.6242454Z STAGE:2023-01-11 21:06:30 17374:17374 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6242524Z ok (2.370s)
2023-01-11T21:18:21.6242530Z 
2023-01-11T21:18:21.6242733Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6242809Z Ran 1 test in 2.370s
2023-01-11T21:18:21.6242814Z 
2023-01-11T21:18:21.6242876Z OK
2023-01-11T21:18:21.6242881Z 
2023-01-11T21:18:21.6242966Z Generating XML reports...
2023-01-11T21:18:21.6243297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210628.xml
2023-01-11T21:18:21.6243537Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6243814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6243940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6244224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6244361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6244366Z 
2023-01-11T21:18:21.6244440Z Running tests...
2023-01-11T21:18:21.6244641Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6244884Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17439
2023-01-11T21:18:21.6245029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17440
2023-01-11T21:18:21.6245186Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17441
2023-01-11T21:18:21.6245512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6245664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6245950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6246086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6246264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6246536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6246658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6246926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6247062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6247243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6247510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6247634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6247915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6248051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6248229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6248515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6248814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6249109Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6249275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6249439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6249597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6249774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6249945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6250117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6250404Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6250703Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6250995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6251402Z STAGE:2023-01-11 21:06:34 17441:17441 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:34 17440:17440 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6251408Z 
2023-01-11T21:18:21.6251655Z STAGE:2023-01-11 21:06:34 17439:17439 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6251901Z STAGE:2023-01-11 21:06:34 17439:17439 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6252174Z STAGE:2023-01-11 21:06:34 17441:17441 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6252461Z STAGE:2023-01-11 21:06:34 17439:17439 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6252708Z STAGE:2023-01-11 21:06:34 17440:17440 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6252964Z STAGE:2023-01-11 21:06:34 17441:17441 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6253206Z STAGE:2023-01-11 21:06:34 17440:17440 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6253446Z STAGE:2023-01-11 21:06:34 17439:17439 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6253683Z STAGE:2023-01-11 21:06:34 17441:17441 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6253918Z STAGE:2023-01-11 21:06:34 17440:17440 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6254163Z STAGE:2023-01-11 21:06:34 17439:17439 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6254405Z STAGE:2023-01-11 21:06:34 17440:17440 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6254643Z STAGE:2023-01-11 21:06:34 17441:17441 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6254899Z STAGE:2023-01-11 21:06:34 17439:17439 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6255155Z STAGE:2023-01-11 21:06:34 17440:17440 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6255396Z STAGE:2023-01-11 21:06:34 17441:17441 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6255637Z STAGE:2023-01-11 21:06:34 17439:17439 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6255873Z STAGE:2023-01-11 21:06:34 17441:17441 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6256111Z STAGE:2023-01-11 21:06:34 17440:17440 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6256523Z STAGE:2023-01-11 21:06:34 17441:17441 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:34 17440:17440 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6256529Z 
2023-01-11T21:18:21.6256775Z STAGE:2023-01-11 21:06:34 17439:17439 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6257202Z STAGE:2023-01-11 21:06:34 17441:17441 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:34 17440:17440 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6257208Z 
2023-01-11T21:18:21.6257462Z STAGE:2023-01-11 21:06:34 17439:17439 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6257530Z ok (2.283s)
2023-01-11T21:18:21.6257538Z 
2023-01-11T21:18:21.6257740Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6257815Z Ran 1 test in 2.283s
2023-01-11T21:18:21.6257821Z 
2023-01-11T21:18:21.6257870Z OK
2023-01-11T21:18:21.6257875Z 
2023-01-11T21:18:21.6257959Z Generating XML reports...
2023-01-11T21:18:21.6258307Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210633.xml
2023-01-11T21:18:21.6258541Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6258820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6258946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6259230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6259366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6259405Z 
2023-01-11T21:18:21.6259465Z Running tests...
2023-01-11T21:18:21.6259694Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6259930Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17504
2023-01-11T21:18:21.6260090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17505
2023-01-11T21:18:21.6260247Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17506
2023-01-11T21:18:21.6260524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6260650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6260940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6261079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6261245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6261550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6261983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6262110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6262397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6262535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6262711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6262991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6263123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6263395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6263532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6263712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6264012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6264311Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6264478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6264650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6264813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6264981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6265157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6265329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6265628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6265924Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6266175Z STAGE:2023-01-11 21:06:38 17505:17505 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6266488Z STAGE:2023-01-11 21:06:38 17506:17506 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6266941Z STAGE:2023-01-11 21:06:38 17505:17505 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:38 17506:17506 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6266949Z 
2023-01-11T21:18:21.6267387Z STAGE:2023-01-11 21:06:38 17506:17506 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:38 17505:17505 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6267393Z 
2023-01-11T21:18:21.6267636Z STAGE:2023-01-11 21:06:38 17506:17506 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6267874Z STAGE:2023-01-11 21:06:38 17505:17505 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6268109Z STAGE:2023-01-11 21:06:38 17506:17506 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6268357Z STAGE:2023-01-11 21:06:38 17505:17505 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6268615Z STAGE:2023-01-11 21:06:38 17506:17506 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6268878Z STAGE:2023-01-11 21:06:38 17505:17505 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6269175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6269244Z ok (2.275s)
2023-01-11T21:18:21.6269250Z 
2023-01-11T21:18:21.6269453Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6269530Z Ran 1 test in 2.275s
2023-01-11T21:18:21.6269536Z 
2023-01-11T21:18:21.6269599Z OK
2023-01-11T21:18:21.6269604Z 
2023-01-11T21:18:21.6269680Z Generating XML reports...
2023-01-11T21:18:21.6270029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210637.xml
2023-01-11T21:18:21.6270262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6270545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6270669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6270950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6271085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6271090Z 
2023-01-11T21:18:21.6271162Z Running tests...
2023-01-11T21:18:21.6271346Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6271580Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17565
2023-01-11T21:18:21.6271743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17566
2023-01-11T21:18:21.6271904Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17567
2023-01-11T21:18:21.6272179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6272302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6272580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6272714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6272891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6273152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6273308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6273619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6273754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6273934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6274235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6274508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6274630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6274908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6275032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6275204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6275506Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6275803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6275968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6276132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6276292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6276463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6276639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6276797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6277095Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6277388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6277637Z STAGE:2023-01-11 21:06:43 17567:17567 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6277878Z STAGE:2023-01-11 21:06:43 17566:17566 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6278288Z STAGE:2023-01-11 21:06:43 17567:17567 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:43 17566:17566 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6278297Z 
2023-01-11T21:18:21.6278727Z STAGE:2023-01-11 21:06:43 17566:17566 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:43 17567:17567 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6278733Z 
2023-01-11T21:18:21.6278971Z STAGE:2023-01-11 21:06:43 17567:17567 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6279207Z STAGE:2023-01-11 21:06:43 17566:17566 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6279456Z STAGE:2023-01-11 21:06:43 17567:17567 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6279686Z STAGE:2023-01-11 21:06:43 17566:17566 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6279944Z STAGE:2023-01-11 21:06:43 17567:17567 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6280201Z STAGE:2023-01-11 21:06:43 17566:17566 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6280531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6280634Z ok (2.274s)
2023-01-11T21:18:21.6280640Z 
2023-01-11T21:18:21.6280839Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6280914Z Ran 1 test in 2.274s
2023-01-11T21:18:21.6280920Z 
2023-01-11T21:18:21.6280979Z OK
2023-01-11T21:18:21.6280985Z 
2023-01-11T21:18:21.6281069Z Generating XML reports...
2023-01-11T21:18:21.6281400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210641.xml
2023-01-11T21:18:21.6281706Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6281988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6282116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6282399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6282535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6282541Z 
2023-01-11T21:18:21.6282615Z Running tests...
2023-01-11T21:18:21.6282815Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6283058Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17626
2023-01-11T21:18:21.6283203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17627
2023-01-11T21:18:21.6283358Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17628
2023-01-11T21:18:21.6283636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6283762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6284049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6284184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6284363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6284636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6284746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6285027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6285163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6285341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6285618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6285742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6286022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6286157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6286331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6286619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6286916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6287244Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6287472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6287638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6287799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6287975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6288149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6288323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6288605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6288906Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6289161Z STAGE:2023-01-11 21:06:47 17627:17627 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6289403Z STAGE:2023-01-11 21:06:47 17628:17628 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6289652Z STAGE:2023-01-11 21:06:47 17627:17627 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6289901Z STAGE:2023-01-11 21:06:47 17628:17628 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6290158Z STAGE:2023-01-11 21:06:47 17627:17627 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6290416Z STAGE:2023-01-11 21:06:47 17628:17628 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6290658Z STAGE:2023-01-11 21:06:47 17627:17627 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6290885Z STAGE:2023-01-11 21:06:47 17628:17628 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6291134Z STAGE:2023-01-11 21:06:47 17628:17628 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6291555Z STAGE:2023-01-11 21:06:47 17627:17627 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:47 17628:17628 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6291561Z 
2023-01-11T21:18:21.6291816Z STAGE:2023-01-11 21:06:47 17627:17627 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6292115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6292183Z ok (2.278s)
2023-01-11T21:18:21.6292189Z 
2023-01-11T21:18:21.6292385Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6292463Z Ran 1 test in 2.278s
2023-01-11T21:18:21.6292469Z 
2023-01-11T21:18:21.6292534Z OK
2023-01-11T21:18:21.6292539Z 
2023-01-11T21:18:21.6292623Z Generating XML reports...
2023-01-11T21:18:21.6292954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210645.xml
2023-01-11T21:18:21.6293187Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6293463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6293587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6293869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6294004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6294010Z 
2023-01-11T21:18:21.6294130Z Running tests...
2023-01-11T21:18:21.6294328Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6294578Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17687
2023-01-11T21:18:21.6294738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17688
2023-01-11T21:18:21.6294895Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17689
2023-01-11T21:18:21.6295170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6295292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6295571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6295704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6295883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6296155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6296266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6296545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6296681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6296859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6297159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6297432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6297557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6297841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6297975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6298138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6298436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6298733Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6298901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6299063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6299227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6299402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6299575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6299735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6300032Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6300323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6300570Z STAGE:2023-01-11 21:06:51 17689:17689 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6300811Z STAGE:2023-01-11 21:06:51 17688:17688 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6301089Z STAGE:2023-01-11 21:06:51 17688:17688 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6301361Z STAGE:2023-01-11 21:06:51 17689:17689 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6301618Z STAGE:2023-01-11 21:06:51 17688:17688 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6301985Z STAGE:2023-01-11 21:06:51 17689:17689 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6302229Z STAGE:2023-01-11 21:06:51 17688:17688 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6302454Z STAGE:2023-01-11 21:06:51 17689:17689 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6302703Z STAGE:2023-01-11 21:06:51 17689:17689 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6302948Z STAGE:2023-01-11 21:06:51 17688:17688 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6303209Z STAGE:2023-01-11 21:06:51 17689:17689 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6303469Z STAGE:2023-01-11 21:06:51 17688:17688 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6303770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6303840Z ok (2.280s)
2023-01-11T21:18:21.6303846Z 
2023-01-11T21:18:21.6304046Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6304108Z Ran 1 test in 2.281s
2023-01-11T21:18:21.6304128Z 
2023-01-11T21:18:21.6304177Z OK
2023-01-11T21:18:21.6304183Z 
2023-01-11T21:18:21.6304267Z Generating XML reports...
2023-01-11T21:18:21.6304612Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210649.xml
2023-01-11T21:18:21.6304849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6305126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6305251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6305537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6305672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6305677Z 
2023-01-11T21:18:21.6305737Z Running tests...
2023-01-11T21:18:21.6305934Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6306164Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17748
2023-01-11T21:18:21.6306321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17749
2023-01-11T21:18:21.6306478Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17750
2023-01-11T21:18:21.6306754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6306877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6307156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6307290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6307454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6307723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6307846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6308185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6308353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6308532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6308806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6308930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6309197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6309332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6309509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6309810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6310113Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6310405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6310573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6310739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6310903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6311137Z STAGE:2023-01-11 21:06:55 17749:17749 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6311378Z STAGE:2023-01-11 21:06:55 17750:17750 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6311623Z STAGE:2023-01-11 21:06:55 17748:17748 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6311873Z STAGE:2023-01-11 21:06:55 17748:17748 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6312116Z STAGE:2023-01-11 21:06:55 17750:17750 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6312374Z STAGE:2023-01-11 21:06:55 17748:17748 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6312619Z STAGE:2023-01-11 21:06:55 17749:17749 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6312871Z STAGE:2023-01-11 21:06:55 17750:17750 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6313123Z STAGE:2023-01-11 21:06:55 17749:17749 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6313349Z STAGE:2023-01-11 21:06:55 17748:17748 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6313587Z STAGE:2023-01-11 21:06:55 17750:17750 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6313823Z STAGE:2023-01-11 21:06:55 17749:17749 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6314397Z STAGE:2023-01-11 21:06:55 17748:17748 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:55 17749:17749 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:55 17750:17750 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6314403Z 
2023-01-11T21:18:21.6314407Z 
2023-01-11T21:18:21.6314664Z STAGE:2023-01-11 21:06:55 17748:17748 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6315090Z STAGE:2023-01-11 21:06:55 17750:17750 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:55 17749:17749 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6315127Z 
2023-01-11T21:18:21.6315369Z STAGE:2023-01-11 21:06:55 17748:17748 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6315633Z STAGE:2023-01-11 21:06:55 17750:17750 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6315872Z STAGE:2023-01-11 21:06:55 17749:17749 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6316118Z STAGE:2023-01-11 21:06:55 17750:17750 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6316359Z STAGE:2023-01-11 21:06:55 17749:17749 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6316787Z STAGE:2023-01-11 21:06:55 17750:17750 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:55 17749:17749 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6316792Z 
2023-01-11T21:18:21.6317024Z STAGE:2023-01-11 21:06:55 17748:17748 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6317286Z STAGE:2023-01-11 21:06:55 17748:17748 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6317353Z ok (2.278s)
2023-01-11T21:18:21.6317359Z 
2023-01-11T21:18:21.6317558Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6317633Z Ran 1 test in 2.278s
2023-01-11T21:18:21.6317638Z 
2023-01-11T21:18:21.6317699Z OK
2023-01-11T21:18:21.6317704Z 
2023-01-11T21:18:21.6317788Z Generating XML reports...
2023-01-11T21:18:21.6318132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210653.xml
2023-01-11T21:18:21.6318364Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6318627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6318754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6319037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6319172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6319177Z 
2023-01-11T21:18:21.6319250Z Running tests...
2023-01-11T21:18:21.6319447Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6319677Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17804
2023-01-11T21:18:21.6319834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17805
2023-01-11T21:18:21.6319976Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17806
2023-01-11T21:18:21.6320252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6320377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6320660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6320795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6320972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6321247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6321371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6321749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6321872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6322052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6322400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6322525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6322807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6322942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6323121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6323422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6323725Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6324011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6324184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6324355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6324518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6324768Z STAGE:2023-01-11 21:06:59 17805:17805 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6325015Z STAGE:2023-01-11 21:06:59 17806:17806 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6325262Z STAGE:2023-01-11 21:06:59 17804:17804 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6325513Z STAGE:2023-01-11 21:06:59 17804:17804 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6325762Z STAGE:2023-01-11 21:06:59 17806:17806 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6325991Z STAGE:2023-01-11 21:06:59 17805:17805 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6326252Z STAGE:2023-01-11 21:06:59 17804:17804 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6326513Z STAGE:2023-01-11 21:06:59 17806:17806 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6326770Z STAGE:2023-01-11 21:06:59 17805:17805 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6327326Z STAGE:2023-01-11 21:06:59 17804:17804 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:59 17806:17806 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:06:59 17805:17805 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6327332Z 
2023-01-11T21:18:21.6327339Z 
2023-01-11T21:18:21.6327589Z STAGE:2023-01-11 21:06:59 17804:17804 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6328000Z STAGE:2023-01-11 21:06:59 17805:17805 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:59 17806:17806 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6328006Z 
2023-01-11T21:18:21.6328262Z STAGE:2023-01-11 21:06:59 17804:17804 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6328689Z STAGE:2023-01-11 21:06:59 17805:17805 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:59 17806:17806 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6328696Z 
2023-01-11T21:18:21.6328941Z STAGE:2023-01-11 21:06:59 17804:17804 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6329188Z STAGE:2023-01-11 21:06:59 17806:17806 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6329461Z STAGE:2023-01-11 21:06:59 17805:17805 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6329723Z STAGE:2023-01-11 21:06:59 17806:17806 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6330138Z STAGE:2023-01-11 21:06:59 17805:17805 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:06:59 17804:17804 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6330145Z 
2023-01-11T21:18:21.6330407Z STAGE:2023-01-11 21:06:59 17806:17806 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6330835Z STAGE:2023-01-11 21:06:59 17805:17805 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:06:59 17804:17804 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6330841Z 
2023-01-11T21:18:21.6330910Z ok (2.292s)
2023-01-11T21:18:21.6330918Z 
2023-01-11T21:18:21.6331119Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6331196Z Ran 1 test in 2.293s
2023-01-11T21:18:21.6331202Z 
2023-01-11T21:18:21.6331267Z OK
2023-01-11T21:18:21.6331272Z 
2023-01-11T21:18:21.6331358Z Generating XML reports...
2023-01-11T21:18:21.6331702Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210657.xml
2023-01-11T21:18:21.6331923Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6332203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6332327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6332611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6332747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6332755Z 
2023-01-11T21:18:21.6332828Z Running tests...
2023-01-11T21:18:21.6333028Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6333267Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17860
2023-01-11T21:18:21.6333426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17861
2023-01-11T21:18:21.6333571Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17862
2023-01-11T21:18:21.6333847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6333971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6334252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6334391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6334568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6334841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6334962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6335228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6335361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6335537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6335805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6335928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6336239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6336399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6336577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6336878Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6337163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6337458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6337622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6337789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6337952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6338047Z skip: CUDA is not available. (2.199s)
2023-01-11T21:18:21.6338053Z 
2023-01-11T21:18:21.6338252Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6338326Z Ran 1 test in 2.199s
2023-01-11T21:18:21.6338331Z 
2023-01-11T21:18:21.6338401Z OK (skipped=1)
2023-01-11T21:18:21.6338407Z 
2023-01-11T21:18:21.6338480Z Generating XML reports...
2023-01-11T21:18:21.6338822Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210701.xml
2023-01-11T21:18:21.6339053Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6339331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6339457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6339737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6339871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6339876Z 
2023-01-11T21:18:21.6339949Z Running tests...
2023-01-11T21:18:21.6340135Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6340379Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17913
2023-01-11T21:18:21.6340535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17914
2023-01-11T21:18:21.6340688Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17915
2023-01-11T21:18:21.6340962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6341087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6341372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6341506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6341785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6342049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6342176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6342460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6342594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6342839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6343149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6343276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6343557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6343692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6343853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6344158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6344457Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6344755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6344922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6345087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6345248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6345347Z skip: CUDA is not available. (2.184s)
2023-01-11T21:18:21.6345352Z 
2023-01-11T21:18:21.6345550Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6345613Z Ran 1 test in 2.185s
2023-01-11T21:18:21.6345619Z 
2023-01-11T21:18:21.6345689Z OK (skipped=1)
2023-01-11T21:18:21.6345694Z 
2023-01-11T21:18:21.6345778Z Generating XML reports...
2023-01-11T21:18:21.6346118Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210705.xml
2023-01-11T21:18:21.6346354Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6346630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6346755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6347035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6347156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6347173Z 
2023-01-11T21:18:21.6347234Z Running tests...
2023-01-11T21:18:21.6347432Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6347664Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17966
2023-01-11T21:18:21.6347823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17967
2023-01-11T21:18:21.6347978Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17968
2023-01-11T21:18:21.6348252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6348375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6348656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6348778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6348956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6349257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6349567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6349689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6349998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6350133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6350309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6350581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6350691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6350968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6351102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6351282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6351586Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6351883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6352052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6352216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6352361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6352611Z STAGE:2023-01-11 21:07:11 17966:17966 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6352854Z STAGE:2023-01-11 21:07:11 17968:17968 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6353091Z STAGE:2023-01-11 21:07:11 17967:17967 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6353343Z STAGE:2023-01-11 21:07:11 17966:17966 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6353588Z STAGE:2023-01-11 21:07:11 17968:17968 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6353830Z STAGE:2023-01-11 21:07:11 17967:17967 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6354087Z STAGE:2023-01-11 21:07:11 17966:17966 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6354343Z STAGE:2023-01-11 21:07:11 17968:17968 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6354585Z STAGE:2023-01-11 21:07:11 17967:17967 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6354828Z STAGE:2023-01-11 21:07:11 17966:17966 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6355227Z STAGE:2023-01-11 21:07:11 17967:17967 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:07:11 17968:17968 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6355233Z 
2023-01-11T21:18:21.6355642Z STAGE:2023-01-11 21:07:11 17968:17968 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:07:11 17966:17966 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6355649Z 
2023-01-11T21:18:21.6355897Z STAGE:2023-01-11 21:07:11 17967:17967 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6356324Z STAGE:2023-01-11 21:07:11 17968:17968 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:07:11 17966:17966 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6356361Z 
2023-01-11T21:18:21.6356616Z STAGE:2023-01-11 21:07:11 17967:17967 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6356887Z STAGE:2023-01-11 21:07:11 17968:17968 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6357132Z STAGE:2023-01-11 21:07:11 17967:17967 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6357370Z STAGE:2023-01-11 21:07:11 17966:17966 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6357614Z STAGE:2023-01-11 21:07:11 17966:17966 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6357856Z STAGE:2023-01-11 21:07:11 17967:17967 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6358081Z STAGE:2023-01-11 21:07:11 17968:17968 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6358335Z STAGE:2023-01-11 21:07:11 17966:17966 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6358594Z STAGE:2023-01-11 21:07:11 17967:17967 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6358846Z STAGE:2023-01-11 21:07:11 17968:17968 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6358914Z ok (2.278s)
2023-01-11T21:18:21.6358920Z 
2023-01-11T21:18:21.6359118Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6359193Z Ran 1 test in 2.278s
2023-01-11T21:18:21.6359199Z 
2023-01-11T21:18:21.6359260Z OK
2023-01-11T21:18:21.6359265Z 
2023-01-11T21:18:21.6359337Z Generating XML reports...
2023-01-11T21:18:21.6359683Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210709.xml
2023-01-11T21:18:21.6359918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6360192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6360318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6360601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6360735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6360741Z 
2023-01-11T21:18:21.6360813Z Running tests...
2023-01-11T21:18:21.6361009Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6361234Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18022
2023-01-11T21:18:21.6361391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18023
2023-01-11T21:18:21.6361548Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18024
2023-01-11T21:18:21.6361898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6362023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6362310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6362445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6362623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6362882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6363007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6363286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6363422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6363640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6363942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6364066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6364345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6364478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6364639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6364938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6365237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6365537Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6365703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6365866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6366031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6366126Z skip: CUDA is not available. (2.176s)
2023-01-11T21:18:21.6366131Z 
2023-01-11T21:18:21.6366329Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6366391Z Ran 1 test in 2.177s
2023-01-11T21:18:21.6366397Z 
2023-01-11T21:18:21.6366469Z OK (skipped=1)
2023-01-11T21:18:21.6366474Z 
2023-01-11T21:18:21.6366872Z Generating XML reports...
2023-01-11T21:18:21.6367262Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210714.xml
2023-01-11T21:18:21.6367688Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6368184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6368315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6368610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6368777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6368784Z 
2023-01-11T21:18:21.6368857Z Running tests...
2023-01-11T21:18:21.6369049Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6369280Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18075
2023-01-11T21:18:21.6369443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18076
2023-01-11T21:18:21.6369601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18077
2023-01-11T21:18:21.6369876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6369999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6370281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6370417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6370582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6370857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6371029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6371341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6371482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6371657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6371955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6372229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6372352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6372620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6372757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6372934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6373235Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6373532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6373697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6373857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6374113Z STAGE:2023-01-11 21:07:19 18077:18077 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6374358Z STAGE:2023-01-11 21:07:19 18076:18076 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6374512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6374760Z STAGE:2023-01-11 21:07:19 18075:18075 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6375007Z STAGE:2023-01-11 21:07:19 18075:18075 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6375269Z STAGE:2023-01-11 21:07:19 18075:18075 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6375517Z STAGE:2023-01-11 21:07:19 18077:18077 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6375761Z STAGE:2023-01-11 21:07:19 18076:18076 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6376014Z STAGE:2023-01-11 21:07:19 18077:18077 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6376269Z STAGE:2023-01-11 21:07:19 18076:18076 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6376511Z STAGE:2023-01-11 21:07:19 18075:18075 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6376737Z STAGE:2023-01-11 21:07:19 18077:18077 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6376970Z STAGE:2023-01-11 21:07:19 18076:18076 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6377216Z STAGE:2023-01-11 21:07:19 18077:18077 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6377459Z STAGE:2023-01-11 21:07:19 18075:18075 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6377715Z STAGE:2023-01-11 21:07:19 18077:18077 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6377956Z STAGE:2023-01-11 21:07:19 18076:18076 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6378207Z STAGE:2023-01-11 21:07:19 18075:18075 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6378491Z STAGE:2023-01-11 21:07:19 18076:18076 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6378759Z STAGE:2023-01-11 21:07:19 18077:18077 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6378984Z STAGE:2023-01-11 21:07:19 18075:18075 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6379218Z STAGE:2023-01-11 21:07:19 18076:18076 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6379462Z STAGE:2023-01-11 21:07:19 18077:18077 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6379718Z STAGE:2023-01-11 21:07:19 18077:18077 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6379962Z STAGE:2023-01-11 21:07:19 18076:18076 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6380208Z STAGE:2023-01-11 21:07:19 18075:18075 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6380467Z STAGE:2023-01-11 21:07:19 18076:18076 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6380723Z STAGE:2023-01-11 21:07:19 18075:18075 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6380792Z ok (2.388s)
2023-01-11T21:18:21.6380797Z 
2023-01-11T21:18:21.6380984Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6381058Z Ran 1 test in 2.388s
2023-01-11T21:18:21.6381064Z 
2023-01-11T21:18:21.6381124Z OK
2023-01-11T21:18:21.6381130Z 
2023-01-11T21:18:21.6381214Z Generating XML reports...
2023-01-11T21:18:21.6381558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210718.xml
2023-01-11T21:18:21.6381896Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6382180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6382309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6382580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6382720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6382726Z 
2023-01-11T21:18:21.6382801Z Running tests...
2023-01-11T21:18:21.6382998Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6383236Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18131
2023-01-11T21:18:21.6383396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18132
2023-01-11T21:18:21.6383553Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18133
2023-01-11T21:18:21.6383830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6383957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6384226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6384361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6384541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6384865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6385136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6385260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6385610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6385784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6385952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6386229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6386353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6386634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6386770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6386949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6387252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6387556Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6387725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6387877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6388042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6388292Z STAGE:2023-01-11 21:07:24 18131:18131 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6388537Z STAGE:2023-01-11 21:07:24 18132:18132 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6388783Z STAGE:2023-01-11 21:07:24 18133:18133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6389038Z STAGE:2023-01-11 21:07:24 18131:18131 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6389290Z STAGE:2023-01-11 21:07:24 18133:18133 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6389534Z STAGE:2023-01-11 21:07:24 18132:18132 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6389795Z STAGE:2023-01-11 21:07:24 18131:18131 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6390039Z STAGE:2023-01-11 21:07:24 18133:18133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6390297Z STAGE:2023-01-11 21:07:24 18132:18132 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6390545Z STAGE:2023-01-11 21:07:24 18132:18132 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6390785Z STAGE:2023-01-11 21:07:24 18133:18133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6391025Z STAGE:2023-01-11 21:07:24 18131:18131 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6391277Z STAGE:2023-01-11 21:07:24 18132:18132 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6391871Z STAGE:2023-01-11 21:07:24 18132:18132 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:07:24 18133:18133 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:07:24 18131:18131 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6391878Z 
2023-01-11T21:18:21.6391883Z 
2023-01-11T21:18:21.6392140Z STAGE:2023-01-11 21:07:24 18133:18133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6392399Z STAGE:2023-01-11 21:07:24 18131:18131 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6392644Z STAGE:2023-01-11 21:07:24 18132:18132 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6392939Z STAGE:2023-01-11 21:07:24 18133:18133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6393191Z STAGE:2023-01-11 21:07:24 18131:18131 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6393441Z STAGE:2023-01-11 21:07:24 18132:18132 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6393687Z STAGE:2023-01-11 21:07:24 18133:18133 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6393930Z STAGE:2023-01-11 21:07:24 18131:18131 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6394186Z STAGE:2023-01-11 21:07:24 18132:18132 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6394438Z STAGE:2023-01-11 21:07:24 18133:18133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6394695Z STAGE:2023-01-11 21:07:24 18131:18131 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6394767Z ok (2.271s)
2023-01-11T21:18:21.6394772Z 
2023-01-11T21:18:21.6394974Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6395040Z Ran 1 test in 2.271s
2023-01-11T21:18:21.6395044Z 
2023-01-11T21:18:21.6395108Z OK
2023-01-11T21:18:21.6395113Z 
2023-01-11T21:18:21.6395200Z Generating XML reports...
2023-01-11T21:18:21.6395547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210722.xml
2023-01-11T21:18:21.6395782Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6396063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6396191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6396473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6396598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6396617Z 
2023-01-11T21:18:21.6396683Z Running tests...
2023-01-11T21:18:21.6396880Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6397123Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18187
2023-01-11T21:18:21.6397279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18188
2023-01-11T21:18:21.6397434Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18189
2023-01-11T21:18:21.6397708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6397831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6398118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6398244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6398421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6398691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6398813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6399092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6399229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6399409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6399680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6399856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6400162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6400300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6400478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6400782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6401083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6401379Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6401549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6401806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6401961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6402214Z STAGE:2023-01-11 21:07:28 18188:18188 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6402456Z STAGE:2023-01-11 21:07:28 18189:18189 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6402704Z STAGE:2023-01-11 21:07:28 18187:18187 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6402954Z STAGE:2023-01-11 21:07:28 18187:18187 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6403204Z STAGE:2023-01-11 21:07:28 18189:18189 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6403454Z STAGE:2023-01-11 21:07:28 18188:18188 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6403717Z STAGE:2023-01-11 21:07:28 18187:18187 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6403974Z STAGE:2023-01-11 21:07:28 18189:18189 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6404219Z STAGE:2023-01-11 21:07:28 18188:18188 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6404461Z STAGE:2023-01-11 21:07:28 18187:18187 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6404866Z STAGE:2023-01-11 21:07:28 18188:18188 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:07:28 18189:18189 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6404872Z 
2023-01-11T21:18:21.6405118Z STAGE:2023-01-11 21:07:28 18188:18188 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6405364Z STAGE:2023-01-11 21:07:28 18187:18187 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6405608Z STAGE:2023-01-11 21:07:28 18189:18189 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6405864Z STAGE:2023-01-11 21:07:28 18188:18188 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6406119Z STAGE:2023-01-11 21:07:28 18187:18187 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6406372Z STAGE:2023-01-11 21:07:28 18189:18189 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6406609Z STAGE:2023-01-11 21:07:28 18188:18188 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6406832Z STAGE:2023-01-11 21:07:28 18187:18187 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6407067Z STAGE:2023-01-11 21:07:28 18189:18189 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6407348Z STAGE:2023-01-11 21:07:28 18188:18188 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6407617Z STAGE:2023-01-11 21:07:28 18189:18189 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6407858Z STAGE:2023-01-11 21:07:28 18187:18187 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6408121Z STAGE:2023-01-11 21:07:28 18188:18188 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6408376Z STAGE:2023-01-11 21:07:28 18189:18189 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6408630Z STAGE:2023-01-11 21:07:28 18187:18187 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6408698Z ok (2.170s)
2023-01-11T21:18:21.6408703Z 
2023-01-11T21:18:21.6408889Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6408966Z Ran 1 test in 2.170s
2023-01-11T21:18:21.6408972Z 
2023-01-11T21:18:21.6409032Z OK
2023-01-11T21:18:21.6409037Z 
2023-01-11T21:18:21.6409121Z Generating XML reports...
2023-01-11T21:18:21.6409470Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210726.xml
2023-01-11T21:18:21.6409704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6409977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6410101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6410369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6410503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6410508Z 
2023-01-11T21:18:21.6410581Z Running tests...
2023-01-11T21:18:21.6410780Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6411018Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18243
2023-01-11T21:18:21.6411176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18244
2023-01-11T21:18:21.6411334Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18245
2023-01-11T21:18:21.6411611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6411733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6412001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6412132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6412309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6412587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6412710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6412992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6413125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6413302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6413572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6413681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6413961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6414126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6414337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6414641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6414940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6415238Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6415402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6415566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6415719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6415816Z skip: CUDA is not available. (2.176s)
2023-01-11T21:18:21.6415822Z 
2023-01-11T21:18:21.6416021Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6447387Z Ran 1 test in 2.176s
2023-01-11T21:18:21.6447404Z 
2023-01-11T21:18:21.6447574Z OK (skipped=1)
2023-01-11T21:18:21.6447581Z 
2023-01-11T21:18:21.6447676Z Generating XML reports...
2023-01-11T21:18:21.6448094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210730.xml
2023-01-11T21:18:21.6448335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6448624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6448753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6449048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6449190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6449197Z 
2023-01-11T21:18:21.6449269Z Running tests...
2023-01-11T21:18:21.6449468Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6449710Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18296
2023-01-11T21:18:21.6449858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18297
2023-01-11T21:18:21.6450013Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18298
2023-01-11T21:18:21.6450287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6450411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6450697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6450835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6451012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6451285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6451395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6451678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6451815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6451992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6452433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6452750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6452878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6453163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6453298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6453461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6453761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6454060Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6454230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6454398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6454563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6454658Z skip: CUDA is not available. (2.177s)
2023-01-11T21:18:21.6454664Z 
2023-01-11T21:18:21.6454864Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6454940Z Ran 1 test in 2.177s
2023-01-11T21:18:21.6454946Z 
2023-01-11T21:18:21.6455005Z OK (skipped=1)
2023-01-11T21:18:21.6455010Z 
2023-01-11T21:18:21.6455097Z Generating XML reports...
2023-01-11T21:18:21.6455443Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210734.xml
2023-01-11T21:18:21.6455679Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6455958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6456085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6456369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6456505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6456510Z 
2023-01-11T21:18:21.6456584Z Running tests...
2023-01-11T21:18:21.6456771Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6457014Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18349
2023-01-11T21:18:21.6457171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18350
2023-01-11T21:18:21.6457329Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18351
2023-01-11T21:18:21.6457608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6457733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6458015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6458153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6458318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6458589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6458713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6458993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6459166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6459372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6459648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6459772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6460053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6460176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6460355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6460657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6460957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6461253Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6461420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6461582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6461890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6461988Z skip: CUDA is not available. (2.174s)
2023-01-11T21:18:21.6461993Z 
2023-01-11T21:18:21.6462184Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6462260Z Ran 1 test in 2.174s
2023-01-11T21:18:21.6462266Z 
2023-01-11T21:18:21.6462341Z OK (skipped=1)
2023-01-11T21:18:21.6462346Z 
2023-01-11T21:18:21.6462432Z Generating XML reports...
2023-01-11T21:18:21.6462782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210738.xml
2023-01-11T21:18:21.6463017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6463295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6463420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6463689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6463825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6463831Z 
2023-01-11T21:18:21.6463904Z Running tests...
2023-01-11T21:18:21.6464102Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6464285Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2023-01-11T21:18:21.6464290Z 
2023-01-11T21:18:21.6464493Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6464569Z Ran 1 test in 0.001s
2023-01-11T21:18:21.6464574Z 
2023-01-11T21:18:21.6464646Z OK (skipped=1)
2023-01-11T21:18:21.6464651Z 
2023-01-11T21:18:21.6464736Z Generating XML reports...
2023-01-11T21:18:21.6465068Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210742.xml
2023-01-11T21:18:21.6465302Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6465582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6465705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6466045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6466215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6466222Z 
2023-01-11T21:18:21.6466296Z Running tests...
2023-01-11T21:18:21.6466495Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6466683Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2023-01-11T21:18:21.6466689Z 
2023-01-11T21:18:21.6466872Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6466948Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6466954Z 
2023-01-11T21:18:21.6467026Z OK (skipped=1)
2023-01-11T21:18:21.6467031Z 
2023-01-11T21:18:21.6467117Z Generating XML reports...
2023-01-11T21:18:21.6467456Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210744.xml
2023-01-11T21:18:21.6467692Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6467970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6468095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6468363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6468500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6468505Z 
2023-01-11T21:18:21.6468580Z Running tests...
2023-01-11T21:18:21.6468776Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6468965Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2023-01-11T21:18:21.6468971Z 
2023-01-11T21:18:21.6469165Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6469242Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6469248Z 
2023-01-11T21:18:21.6469319Z OK (skipped=1)
2023-01-11T21:18:21.6469324Z 
2023-01-11T21:18:21.6469411Z Generating XML reports...
2023-01-11T21:18:21.6469738Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210745.xml
2023-01-11T21:18:21.6469973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6470247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6470372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6470652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6470787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6470794Z 
2023-01-11T21:18:21.6470867Z Running tests...
2023-01-11T21:18:21.6471066Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6471264Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2023-01-11T21:18:21.6471269Z 
2023-01-11T21:18:21.6471450Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6471527Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6471532Z 
2023-01-11T21:18:21.6471604Z OK (skipped=1)
2023-01-11T21:18:21.6471609Z 
2023-01-11T21:18:21.6471693Z Generating XML reports...
2023-01-11T21:18:21.6472032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210747.xml
2023-01-11T21:18:21.6472266Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6472537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6472718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6473027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6473151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6473157Z 
2023-01-11T21:18:21.6473234Z Running tests...
2023-01-11T21:18:21.6473430Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6473617Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2023-01-11T21:18:21.6473623Z 
2023-01-11T21:18:21.6473826Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6473903Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6473909Z 
2023-01-11T21:18:21.6473981Z OK (skipped=1)
2023-01-11T21:18:21.6473986Z 
2023-01-11T21:18:21.6474074Z Generating XML reports...
2023-01-11T21:18:21.6474399Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210749.xml
2023-01-11T21:18:21.6474633Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6474908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6475032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6475313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6475449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6475455Z 
2023-01-11T21:18:21.6475530Z Running tests...
2023-01-11T21:18:21.6475724Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6475924Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2023-01-11T21:18:21.6475933Z 
2023-01-11T21:18:21.6476117Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6476193Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6476198Z 
2023-01-11T21:18:21.6476272Z OK (skipped=1)
2023-01-11T21:18:21.6476277Z 
2023-01-11T21:18:21.6476361Z Generating XML reports...
2023-01-11T21:18:21.6476704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210750.xml
2023-01-11T21:18:21.6476935Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6477211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6477338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6477623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6477747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6477752Z 
2023-01-11T21:18:21.6477826Z Running tests...
2023-01-11T21:18:21.6478024Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6478206Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2023-01-11T21:18:21.6478211Z 
2023-01-11T21:18:21.6478403Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6478477Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6478483Z 
2023-01-11T21:18:21.6478554Z OK (skipped=1)
2023-01-11T21:18:21.6478559Z 
2023-01-11T21:18:21.6478643Z Generating XML reports...
2023-01-11T21:18:21.6478971Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210752.xml
2023-01-11T21:18:21.6479203Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6479538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6479663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6479946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6480081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6480087Z 
2023-01-11T21:18:21.6480160Z Running tests...
2023-01-11T21:18:21.6480357Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6480557Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2023-01-11T21:18:21.6480562Z 
2023-01-11T21:18:21.6480744Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6480825Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6480829Z 
2023-01-11T21:18:21.6480902Z OK (skipped=1)
2023-01-11T21:18:21.6480907Z 
2023-01-11T21:18:21.6480993Z Generating XML reports...
2023-01-11T21:18:21.6481337Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210754.xml
2023-01-11T21:18:21.6481648Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6481940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6482066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6482350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6482473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6482478Z 
2023-01-11T21:18:21.6482558Z Running tests...
2023-01-11T21:18:21.6482757Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6482967Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:18:21.6482973Z 
2023-01-11T21:18:21.6483169Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6483245Z Ran 1 test in 0.001s
2023-01-11T21:18:21.6483251Z 
2023-01-11T21:18:21.6483323Z OK (skipped=1)
2023-01-11T21:18:21.6483328Z 
2023-01-11T21:18:21.6483411Z Generating XML reports...
2023-01-11T21:18:21.6483752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210755.xml
2023-01-11T21:18:21.6483970Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6484245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6484371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6484655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6484793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6484798Z 
2023-01-11T21:18:21.6484873Z Running tests...
2023-01-11T21:18:21.6485073Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6485288Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:18:21.6485294Z 
2023-01-11T21:18:21.6485489Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6485552Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6485557Z 
2023-01-11T21:18:21.6485630Z OK (skipped=1)
2023-01-11T21:18:21.6485635Z 
2023-01-11T21:18:21.6485761Z Generating XML reports...
2023-01-11T21:18:21.6486100Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210757.xml
2023-01-11T21:18:21.6486359Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6486637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6486762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6487043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6487165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6487183Z 
2023-01-11T21:18:21.6487245Z Running tests...
2023-01-11T21:18:21.6487441Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6487656Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2023-01-11T21:18:21.6487663Z 
2023-01-11T21:18:21.6487859Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6487935Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6487940Z 
2023-01-11T21:18:21.6488012Z OK (skipped=1)
2023-01-11T21:18:21.6488017Z 
2023-01-11T21:18:21.6488101Z Generating XML reports...
2023-01-11T21:18:21.6488442Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210759.xml
2023-01-11T21:18:21.6488660Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6488935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6489059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6489340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6489479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6489484Z 
2023-01-11T21:18:21.6489559Z Running tests...
2023-01-11T21:18:21.6489755Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6489977Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2023-01-11T21:18:21.6489982Z 
2023-01-11T21:18:21.6490176Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6490240Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6490246Z 
2023-01-11T21:18:21.6490322Z OK (skipped=1)
2023-01-11T21:18:21.6490327Z 
2023-01-11T21:18:21.6490410Z Generating XML reports...
2023-01-11T21:18:21.6490752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210800.xml
2023-01-11T21:18:21.6490990Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6491265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6491390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6491674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6491810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6491815Z 
2023-01-11T21:18:21.6491875Z Running tests...
2023-01-11T21:18:21.6492073Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6492290Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:18:21.6492295Z 
2023-01-11T21:18:21.6492517Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6492593Z Ran 1 test in 0.001s
2023-01-11T21:18:21.6492598Z 
2023-01-11T21:18:21.6492669Z OK (skipped=1)
2023-01-11T21:18:21.6492702Z 
2023-01-11T21:18:21.6492788Z Generating XML reports...
2023-01-11T21:18:21.6493129Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210802.xml
2023-01-11T21:18:21.6493361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6493626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6493749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6494028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6494164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6494172Z 
2023-01-11T21:18:21.6494245Z Running tests...
2023-01-11T21:18:21.6494440Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6494672Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:18:21.6494679Z 
2023-01-11T21:18:21.6494868Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6494930Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6494948Z 
2023-01-11T21:18:21.6495008Z OK (skipped=1)
2023-01-11T21:18:21.6495013Z 
2023-01-11T21:18:21.6495097Z Generating XML reports...
2023-01-11T21:18:21.6495433Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210804.xml
2023-01-11T21:18:21.6495667Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6495939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6496063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6496346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6496482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6496488Z 
2023-01-11T21:18:21.6496548Z Running tests...
2023-01-11T21:18:21.6496743Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6496955Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:18:21.6496960Z 
2023-01-11T21:18:21.6497152Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6497228Z Ran 1 test in 0.001s
2023-01-11T21:18:21.6497233Z 
2023-01-11T21:18:21.6497307Z OK (skipped=1)
2023-01-11T21:18:21.6497312Z 
2023-01-11T21:18:21.6497396Z Generating XML reports...
2023-01-11T21:18:21.6497734Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210805.xml
2023-01-11T21:18:21.6497965Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6498222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6498349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6498629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6498764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6498769Z 
2023-01-11T21:18:21.6498842Z Running tests...
2023-01-11T21:18:21.6499036Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6499288Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2023-01-11T21:18:21.6499320Z 
2023-01-11T21:18:21.6499513Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6499589Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6499594Z 
2023-01-11T21:18:21.6499653Z OK (skipped=1)
2023-01-11T21:18:21.6499658Z 
2023-01-11T21:18:21.6499743Z Generating XML reports...
2023-01-11T21:18:21.6500081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210807.xml
2023-01-11T21:18:21.6500312Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6500591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6500714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6501001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6501138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6501143Z 
2023-01-11T21:18:21.6501217Z Running tests...
2023-01-11T21:18:21.6501420Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6501626Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:18:21.6501632Z 
2023-01-11T21:18:21.6501952Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6502030Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6502035Z 
2023-01-11T21:18:21.6502108Z OK (skipped=1)
2023-01-11T21:18:21.6502113Z 
2023-01-11T21:18:21.6502199Z Generating XML reports...
2023-01-11T21:18:21.6502545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210809.xml
2023-01-11T21:18:21.6502779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6503042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6503166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6503446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6503582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6503588Z 
2023-01-11T21:18:21.6503662Z Running tests...
2023-01-11T21:18:21.6503858Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6504077Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:18:21.6504085Z 
2023-01-11T21:18:21.6504280Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6504356Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6504363Z 
2023-01-11T21:18:21.6504423Z OK (skipped=1)
2023-01-11T21:18:21.6504442Z 
2023-01-11T21:18:21.6504515Z Generating XML reports...
2023-01-11T21:18:21.6504855Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210810.xml
2023-01-11T21:18:21.6505090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6505368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6505492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6505775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6505972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6505977Z 
2023-01-11T21:18:21.6506051Z Running tests...
2023-01-11T21:18:21.6506270Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6506489Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2023-01-11T21:18:21.6506495Z 
2023-01-11T21:18:21.6506688Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6506766Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6506771Z 
2023-01-11T21:18:21.6506844Z OK (skipped=1)
2023-01-11T21:18:21.6506848Z 
2023-01-11T21:18:21.6506932Z Generating XML reports...
2023-01-11T21:18:21.6507271Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210812.xml
2023-01-11T21:18:21.6507504Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6507785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6507899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6508179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6508318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6508323Z 
2023-01-11T21:18:21.6508397Z Running tests...
2023-01-11T21:18:21.6508594Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6508818Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2023-01-11T21:18:21.6508825Z 
2023-01-11T21:18:21.6509018Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6509096Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6509102Z 
2023-01-11T21:18:21.6509172Z OK (skipped=1)
2023-01-11T21:18:21.6509178Z 
2023-01-11T21:18:21.6509251Z Generating XML reports...
2023-01-11T21:18:21.6509592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210814.xml
2023-01-11T21:18:21.6509824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6510099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6510223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6510506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6510643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6510648Z 
2023-01-11T21:18:21.6510720Z Running tests...
2023-01-11T21:18:21.6510906Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6511129Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:18:21.6511134Z 
2023-01-11T21:18:21.6511327Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6511403Z Ran 1 test in 0.001s
2023-01-11T21:18:21.6511408Z 
2023-01-11T21:18:21.6511480Z OK (skipped=1)
2023-01-11T21:18:21.6511485Z 
2023-01-11T21:18:21.6511570Z Generating XML reports...
2023-01-11T21:18:21.6511912Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210816.xml
2023-01-11T21:18:21.6512143Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6512417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6512558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6512894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6513033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6513038Z 
2023-01-11T21:18:21.6513112Z Running tests...
2023-01-11T21:18:21.6513309Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6513540Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:18:21.6513546Z 
2023-01-11T21:18:21.6513740Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6513819Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6513824Z 
2023-01-11T21:18:21.6513897Z OK (skipped=1)
2023-01-11T21:18:21.6513902Z 
2023-01-11T21:18:21.6513975Z Generating XML reports...
2023-01-11T21:18:21.6514312Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210817.xml
2023-01-11T21:18:21.6514547Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6514825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6514955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6515241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6515379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6515385Z 
2023-01-11T21:18:21.6515461Z Running tests...
2023-01-11T21:18:21.6515658Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6515860Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2023-01-11T21:18:21.6515868Z 
2023-01-11T21:18:21.6516067Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6516142Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6516148Z 
2023-01-11T21:18:21.6516219Z OK (skipped=1)
2023-01-11T21:18:21.6516225Z 
2023-01-11T21:18:21.6516308Z Generating XML reports...
2023-01-11T21:18:21.6516645Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210819.xml
2023-01-11T21:18:21.6516877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6517151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6517275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6517543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6517681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6517686Z 
2023-01-11T21:18:21.6517764Z Running tests...
2023-01-11T21:18:21.6517957Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6518177Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2023-01-11T21:18:21.6518183Z 
2023-01-11T21:18:21.6518380Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6518455Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6518460Z 
2023-01-11T21:18:21.6518531Z OK (skipped=1)
2023-01-11T21:18:21.6518536Z 
2023-01-11T21:18:21.6518607Z Generating XML reports...
2023-01-11T21:18:21.6518949Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210821.xml
2023-01-11T21:18:21.6519232Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6519535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6519659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6519939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6520077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6520082Z 
2023-01-11T21:18:21.6520157Z Running tests...
2023-01-11T21:18:21.6520353Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6520577Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18642
2023-01-11T21:18:21.6520737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18643
2023-01-11T21:18:21.6520896Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18644
2023-01-11T21:18:21.6521172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6521294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6521655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6521797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6521978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6522253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6522363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6522647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6522787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6522968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6523271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6523545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6523672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6523950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6524071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6524250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6524554Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6524853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6525019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6525185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6525347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6525451Z skip: Need at least 2 CUDA devices (2.184s)
2023-01-11T21:18:21.6525457Z 
2023-01-11T21:18:21.6525655Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6525718Z Ran 1 test in 2.184s
2023-01-11T21:18:21.6525771Z 
2023-01-11T21:18:21.6525831Z OK (skipped=1)
2023-01-11T21:18:21.6525836Z 
2023-01-11T21:18:21.6525921Z Generating XML reports...
2023-01-11T21:18:21.6526291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210822.xml
2023-01-11T21:18:21.6526525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6526799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6526923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6527201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6527336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6527342Z 
2023-01-11T21:18:21.6527404Z Running tests...
2023-01-11T21:18:21.6527605Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6527927Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.001s)
2023-01-11T21:18:21.6527933Z 
2023-01-11T21:18:21.6528128Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6528206Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6528211Z 
2023-01-11T21:18:21.6528283Z OK (skipped=1)
2023-01-11T21:18:21.6528288Z 
2023-01-11T21:18:21.6528373Z Generating XML reports...
2023-01-11T21:18:21.6528711Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210826.xml
2023-01-11T21:18:21.6528945Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6529204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6529330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6529616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6529755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6529761Z 
2023-01-11T21:18:21.6529833Z Running tests...
2023-01-11T21:18:21.6530028Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6530338Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s)
2023-01-11T21:18:21.6530344Z 
2023-01-11T21:18:21.6530535Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6530611Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6530616Z 
2023-01-11T21:18:21.6530676Z OK (skipped=1)
2023-01-11T21:18:21.6530681Z 
2023-01-11T21:18:21.6530764Z Generating XML reports...
2023-01-11T21:18:21.6531104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210828.xml
2023-01-11T21:18:21.6531336Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6531614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6531736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6532016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6532154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6532160Z 
2023-01-11T21:18:21.6532219Z Running tests...
2023-01-11T21:18:21.6532419Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6532641Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18715
2023-01-11T21:18:21.6532830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18716
2023-01-11T21:18:21.6533045Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18717
2023-01-11T21:18:21.6533323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6533450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6533735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6533872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6534038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6534311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6534437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6534722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6534858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6535037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6535310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6535440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6535708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6535845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6536022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6536329Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6536631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6536926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6537092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6537258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6537421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6537478Z ok (3.374s)
2023-01-11T21:18:21.6537497Z 
2023-01-11T21:18:21.6537685Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6537762Z Ran 1 test in 3.374s
2023-01-11T21:18:21.6537769Z 
2023-01-11T21:18:21.6537831Z OK
2023-01-11T21:18:21.6537837Z 
2023-01-11T21:18:21.6537925Z Generating XML reports...
2023-01-11T21:18:21.6538269Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210830.xml
2023-01-11T21:18:21.6538502Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6538776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6538899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6539170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6539307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6539343Z 
2023-01-11T21:18:21.6539416Z Running tests...
2023-01-11T21:18:21.6539612Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6539871Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18768
2023-01-11T21:18:21.6540029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18769
2023-01-11T21:18:21.6540185Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18770
2023-01-11T21:18:21.6540458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6540571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6540854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6540988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6541166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6541441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6541564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6541955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6542093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6542270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6542648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6542922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6543046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6543326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6543460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6543638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6543934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6544230Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6544396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6544548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6544711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6544812Z skip: CUDA is not available. (2.181s)
2023-01-11T21:18:21.6544820Z 
2023-01-11T21:18:21.6545017Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6545092Z Ran 1 test in 2.182s
2023-01-11T21:18:21.6545097Z 
2023-01-11T21:18:21.6545169Z OK (skipped=1)
2023-01-11T21:18:21.6545174Z 
2023-01-11T21:18:21.6545260Z Generating XML reports...
2023-01-11T21:18:21.6545600Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210835.xml
2023-01-11T21:18:21.6545820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6546095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6546221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6546568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6546741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6546746Z 
2023-01-11T21:18:21.6546822Z Running tests...
2023-01-11T21:18:21.6547023Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6547257Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18821
2023-01-11T21:18:21.6547415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18822
2023-01-11T21:18:21.6547556Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18823
2023-01-11T21:18:21.6547833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6547961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6548247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6548381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6548558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6548831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6548955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6549239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6549362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6549541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6549817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6549945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6550226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6550361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6550538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6550838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6551120Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6551419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6551589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6551754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6551921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6552095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6552268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6552441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6552738Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6553068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6553379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6553450Z ok (3.480s)
2023-01-11T21:18:21.6553455Z 
2023-01-11T21:18:21.6553651Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6553728Z Ran 1 test in 3.480s
2023-01-11T21:18:21.6553733Z 
2023-01-11T21:18:21.6553794Z OK
2023-01-11T21:18:21.6553800Z 
2023-01-11T21:18:21.6553884Z Generating XML reports...
2023-01-11T21:18:21.6554223Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210839.xml
2023-01-11T21:18:21.6554459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6554721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6554849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6555132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6555269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6555275Z 
2023-01-11T21:18:21.6555349Z Running tests...
2023-01-11T21:18:21.6555546Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6555785Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18883
2023-01-11T21:18:21.6555943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18884
2023-01-11T21:18:21.6556097Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18885
2023-01-11T21:18:21.6556356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6556482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6556768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6556905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6557082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6557355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6557479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6557765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6557888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6558068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6558342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6558466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6558746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6558880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6559059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6559359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6559657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6559971Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6560166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6560333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6560496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6560594Z skip: CUDA is not available. (2.173s)
2023-01-11T21:18:21.6560599Z 
2023-01-11T21:18:21.6560799Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6560875Z Ran 1 test in 2.173s
2023-01-11T21:18:21.6560880Z 
2023-01-11T21:18:21.6560952Z OK (skipped=1)
2023-01-11T21:18:21.6560956Z 
2023-01-11T21:18:21.6561041Z Generating XML reports...
2023-01-11T21:18:21.6561371Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210844.xml
2023-01-11T21:18:21.6561676Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6561963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6562090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6562374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6562512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6562517Z 
2023-01-11T21:18:21.6562592Z Running tests...
2023-01-11T21:18:21.6562790Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6563004Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18936
2023-01-11T21:18:21.6563166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18937
2023-01-11T21:18:21.6563323Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18938
2023-01-11T21:18:21.6563599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6563724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6564005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6564139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6564318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6564592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6564703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6564990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6565126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6565303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6565573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6565700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6565978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6566112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6566288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6566626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6566958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6567257Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6567421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6567590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6567752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6567928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6568104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6568278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6568566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6568864Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6569155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6569225Z ok (3.081s)
2023-01-11T21:18:21.6569231Z 
2023-01-11T21:18:21.6569433Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6569510Z Ran 1 test in 3.081s
2023-01-11T21:18:21.6569516Z 
2023-01-11T21:18:21.6569578Z OK
2023-01-11T21:18:21.6569583Z 
2023-01-11T21:18:21.6569670Z Generating XML reports...
2023-01-11T21:18:21.6570002Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210848.xml
2023-01-11T21:18:21.6570241Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6570513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6570637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6570920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6571056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6571062Z 
2023-01-11T21:18:21.6571135Z Running tests...
2023-01-11T21:18:21.6571330Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6571564Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18995
2023-01-11T21:18:21.6571715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18996
2023-01-11T21:18:21.6571874Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18997
2023-01-11T21:18:21.6572146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6572272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6572554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6572689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6572867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6573138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6573298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6573592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6573730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6573907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6574177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6574300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6574583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6574719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6574896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6575187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6575486Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6575777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6575944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6576107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6576270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6576368Z skip: CUDA is not available. (2.185s)
2023-01-11T21:18:21.6576376Z 
2023-01-11T21:18:21.6576573Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6576648Z Ran 1 test in 2.185s
2023-01-11T21:18:21.6576653Z 
2023-01-11T21:18:21.6576714Z OK (skipped=1)
2023-01-11T21:18:21.6576732Z 
2023-01-11T21:18:21.6576805Z Generating XML reports...
2023-01-11T21:18:21.6577148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210853.xml
2023-01-11T21:18:21.6577380Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6577657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6577782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6578062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6578199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6578204Z 
2023-01-11T21:18:21.6578277Z Running tests...
2023-01-11T21:18:21.6578462Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6578702Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19048
2023-01-11T21:18:21.6578862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19049
2023-01-11T21:18:21.6579017Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19050
2023-01-11T21:18:21.6579292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6579417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6579701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6579874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6580072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6580346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6580468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6580750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6580884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6581059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6581358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6581630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6581879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6582149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6582283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6582461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6582759Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6583057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6583222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6583389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6583567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6583805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6583968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6584143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6584445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6584737Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6585029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6585101Z ok (3.276s)
2023-01-11T21:18:21.6585110Z 
2023-01-11T21:18:21.6585313Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6585389Z Ran 1 test in 3.276s
2023-01-11T21:18:21.6585394Z 
2023-01-11T21:18:21.6585456Z OK
2023-01-11T21:18:21.6585461Z 
2023-01-11T21:18:21.6585532Z Generating XML reports...
2023-01-11T21:18:21.6585878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210857.xml
2023-01-11T21:18:21.6586111Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6586386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6586510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6586796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6586995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6587000Z 
2023-01-11T21:18:21.6587111Z Running tests...
2023-01-11T21:18:21.6587312Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6587537Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19110
2023-01-11T21:18:21.6587695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19111
2023-01-11T21:18:21.6587849Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19112
2023-01-11T21:18:21.6588127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6588251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6588538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6588675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6588854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6589113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6589240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6589521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6589657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6589835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6590108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6590232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6590517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6590653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6590819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6591119Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6591419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6591717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6591885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6592051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6592214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6592389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6592562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6592845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6593017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6593314Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6593644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6593713Z ok (3.381s)
2023-01-11T21:18:21.6593746Z 
2023-01-11T21:18:21.6593947Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6594024Z Ran 1 test in 3.381s
2023-01-11T21:18:21.6594030Z 
2023-01-11T21:18:21.6594090Z OK
2023-01-11T21:18:21.6594095Z 
2023-01-11T21:18:21.6594180Z Generating XML reports...
2023-01-11T21:18:21.6594509Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210902.xml
2023-01-11T21:18:21.6594741Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6595014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6595138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6595424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6595564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6595569Z 
2023-01-11T21:18:21.6595642Z Running tests...
2023-01-11T21:18:21.6595840Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6596063Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19172
2023-01-11T21:18:21.6596223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19173
2023-01-11T21:18:21.6596379Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19174
2023-01-11T21:18:21.6596655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6596781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6597062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6597200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6597375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6597647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6597757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6598041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6598175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6598352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6598655Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6598934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6599055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6599336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6599470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6599633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6599935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6600231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6600429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6600622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6600785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6600960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6601133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6601306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6601670Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6601972Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6602272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6602342Z ok (7.180s)
2023-01-11T21:18:21.6602348Z 
2023-01-11T21:18:21.6602548Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6602625Z Ran 1 test in 7.180s
2023-01-11T21:18:21.6602631Z 
2023-01-11T21:18:21.6602694Z OK
2023-01-11T21:18:21.6602699Z 
2023-01-11T21:18:21.6602784Z Generating XML reports...
2023-01-11T21:18:21.6603110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210908.xml
2023-01-11T21:18:21.6603344Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6603618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6603745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6604030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6604165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6604171Z 
2023-01-11T21:18:21.6604245Z Running tests...
2023-01-11T21:18:21.6604444Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6604681Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19231
2023-01-11T21:18:21.6604826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19232
2023-01-11T21:18:21.6604986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19233
2023-01-11T21:18:21.6605256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6605382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6605666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6605800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6605978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6606252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6606374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6606645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6606778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6606996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6607324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6607596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6607719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6608001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6608140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6608301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6608598Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6608900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6609068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6609233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6609395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6609464Z ok (2.274s)
2023-01-11T21:18:21.6609469Z 
2023-01-11T21:18:21.6609667Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6609742Z Ran 1 test in 2.274s
2023-01-11T21:18:21.6609748Z 
2023-01-11T21:18:21.6609796Z OK
2023-01-11T21:18:21.6609800Z 
2023-01-11T21:18:21.6609885Z Generating XML reports...
2023-01-11T21:18:21.6610225Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210917.xml
2023-01-11T21:18:21.6610460Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6610742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6610867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6611149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6611283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6611288Z 
2023-01-11T21:18:21.6611362Z Running tests...
2023-01-11T21:18:21.6611547Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6611787Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19284
2023-01-11T21:18:21.6611946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19285
2023-01-11T21:18:21.6612100Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19286
2023-01-11T21:18:21.6612376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6612499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6612782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6612917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6613081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6613355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6613476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6613801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6613965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6614146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6614416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6614540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6614818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6614941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6615115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6615425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6615730Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6616022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6616188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6616351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6616512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6616578Z ok (2.281s)
2023-01-11T21:18:21.6616583Z 
2023-01-11T21:18:21.6616770Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6616851Z Ran 1 test in 2.281s
2023-01-11T21:18:21.6616856Z 
2023-01-11T21:18:21.6616919Z OK
2023-01-11T21:18:21.6616924Z 
2023-01-11T21:18:21.6617009Z Generating XML reports...
2023-01-11T21:18:21.6617358Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210921.xml
2023-01-11T21:18:21.6617593Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6617868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6617994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6618264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6618400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6618405Z 
2023-01-11T21:18:21.6618480Z Running tests...
2023-01-11T21:18:21.6618682Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6618887Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:18:21.6618893Z 
2023-01-11T21:18:21.6619092Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6619169Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6619174Z 
2023-01-11T21:18:21.6619246Z OK (skipped=1)
2023-01-11T21:18:21.6619251Z 
2023-01-11T21:18:21.6619335Z Generating XML reports...
2023-01-11T21:18:21.6619661Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210925.xml
2023-01-11T21:18:21.6619892Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6620164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6620324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6620607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6620772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6620777Z 
2023-01-11T21:18:21.6620851Z Running tests...
2023-01-11T21:18:21.6621047Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6621222Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:18:21.6621243Z 
2023-01-11T21:18:21.6621424Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6621499Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6621503Z 
2023-01-11T21:18:21.6621575Z OK (skipped=1)
2023-01-11T21:18:21.6621580Z 
2023-01-11T21:18:21.6621891Z Generating XML reports...
2023-01-11T21:18:21.6622237Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210926.xml
2023-01-11T21:18:21.6622473Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6622755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6622882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6623151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6623287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6623292Z 
2023-01-11T21:18:21.6623369Z Running tests...
2023-01-11T21:18:21.6623561Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6623758Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:18:21.6623766Z 
2023-01-11T21:18:21.6623961Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6624039Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6624044Z 
2023-01-11T21:18:21.6624121Z OK (skipped=1)
2023-01-11T21:18:21.6624126Z 
2023-01-11T21:18:21.6624211Z Generating XML reports...
2023-01-11T21:18:21.6624538Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210928.xml
2023-01-11T21:18:21.6624775Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6625049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6625174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6625457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6625596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6625601Z 
2023-01-11T21:18:21.6625675Z Running tests...
2023-01-11T21:18:21.6625874Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6626062Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:18:21.6626068Z 
2023-01-11T21:18:21.6626249Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6626325Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6626330Z 
2023-01-11T21:18:21.6626402Z OK (skipped=1)
2023-01-11T21:18:21.6626406Z 
2023-01-11T21:18:21.6626491Z Generating XML reports...
2023-01-11T21:18:21.6626838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210930.xml
2023-01-11T21:18:21.6627069Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6627402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6627562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6627836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6627975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6627981Z 
2023-01-11T21:18:21.6628057Z Running tests...
2023-01-11T21:18:21.6628252Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6628445Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.001s)
2023-01-11T21:18:21.6628452Z 
2023-01-11T21:18:21.6628646Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6628722Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6628727Z 
2023-01-11T21:18:21.6628801Z OK (skipped=1)
2023-01-11T21:18:21.6628805Z 
2023-01-11T21:18:21.6628890Z Generating XML reports...
2023-01-11T21:18:21.6629222Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210932.xml
2023-01-11T21:18:21.6629458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6629731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6629855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6630134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6630270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6630275Z 
2023-01-11T21:18:21.6630350Z Running tests...
2023-01-11T21:18:21.6630545Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6630752Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:18:21.6630757Z 
2023-01-11T21:18:21.6630943Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6631020Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6631024Z 
2023-01-11T21:18:21.6631096Z OK (skipped=1)
2023-01-11T21:18:21.6631101Z 
2023-01-11T21:18:21.6631185Z Generating XML reports...
2023-01-11T21:18:21.6631525Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210933.xml
2023-01-11T21:18:21.6631758Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6632031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6632154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6632440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6632563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6632568Z 
2023-01-11T21:18:21.6632643Z Running tests...
2023-01-11T21:18:21.6632842Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6633033Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:18:21.6633039Z 
2023-01-11T21:18:21.6633233Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6633309Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6633314Z 
2023-01-11T21:18:21.6633386Z OK (skipped=1)
2023-01-11T21:18:21.6633391Z 
2023-01-11T21:18:21.6633476Z Generating XML reports...
2023-01-11T21:18:21.6633803Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210935.xml
2023-01-11T21:18:21.6634066Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6634365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6634489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6634768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6634904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6634909Z 
2023-01-11T21:18:21.6634980Z Running tests...
2023-01-11T21:18:21.6635177Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6635372Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:18:21.6635380Z 
2023-01-11T21:18:21.6635562Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6635639Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6635644Z 
2023-01-11T21:18:21.6635714Z OK (skipped=1)
2023-01-11T21:18:21.6635720Z 
2023-01-11T21:18:21.6635804Z Generating XML reports...
2023-01-11T21:18:21.6636141Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210937.xml
2023-01-11T21:18:21.6636373Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6636649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6636773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6637054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6637176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6637183Z 
2023-01-11T21:18:21.6637261Z Running tests...
2023-01-11T21:18:21.6637459Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6637686Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19417
2023-01-11T21:18:21.6637843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19418
2023-01-11T21:18:21.6637997Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19419
2023-01-11T21:18:21.6638271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6638392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6638660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6638796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6638973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6639276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6639545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6639668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6639949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6640083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6640261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6640520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6640714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6641018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6641156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6641334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6641705Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6642008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6642176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6642343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6642493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6642750Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6642994Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6643230Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6643645Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6643651Z 
2023-01-11T21:18:21.6644073Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6644081Z 
2023-01-11T21:18:21.6644342Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6644602Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6644843Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6645245Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6645251Z 
2023-01-11T21:18:21.6645505Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6645753Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6646165Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6646173Z 
2023-01-11T21:18:21.6646612Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6646619Z 
2023-01-11T21:18:21.6646861Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6647102Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6647338Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6647586Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6647864Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6648153Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6648402Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6648659Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6648901Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6649143Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6649383Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6649618Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6649867Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6650125Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6650369Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6650624Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6650869Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6651108Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6651347Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6651586Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6651824Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6652074Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6652496Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6652501Z 
2023-01-11T21:18:21.6652750Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6653007Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6653264Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6653511Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6653740Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6653985Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6654227Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6654466Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6654707Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6654965Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6655220Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6655506Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6655776Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6655999Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6656235Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6656477Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6656733Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6657144Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6657152Z 
2023-01-11T21:18:21.6657582Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6657587Z 
2023-01-11T21:18:21.6657832Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6658070Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6658306Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6658553Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6658799Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6659049Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6659306Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6659554Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6659808Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6660050Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6660288Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6660524Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6660770Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6661003Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6661260Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6661506Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6661885Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6662141Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6662383Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6662623Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6662863Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6663172Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6663614Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6663636Z 
2023-01-11T21:18:21.6663872Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6664131Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6664386Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6664629Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6664869Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6665109Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6665356Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6665773Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6665778Z 
2023-01-11T21:18:21.6666034Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6666279Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6666520Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6666766Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6667007Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6667243Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6667489Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6667744Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6667988Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6668243Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6668490Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6668732Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6668972Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6669212Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6669448Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6669694Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6669936Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6670189Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6670441Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6670717Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6670983Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6671221Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6671456Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6671689Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6671935Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6672176Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6672430Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6672856Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6672862Z 
2023-01-11T21:18:21.6673118Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6673345Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6673590Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6673827Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6674074Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6674328Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6674575Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6674829Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6675073Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6675326Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6675553Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6675787Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6676028Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6676273Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6676516Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6676935Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6676943Z 
2023-01-11T21:18:21.6677197Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6677452Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6677695Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6677935Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6678157Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6678437Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6678719Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6678976Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6679222Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6679475Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6679727Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6679966Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6680205Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6680431Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6680679Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6680933Z STAGE:2023-01-11 21:09:40 19417:19417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6681177Z STAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6681665Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:40 19419:19419 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6681674Z 
2023-01-11T21:18:21.6681934Z STAGE:2023-01-11 21:09:40 19418:19418 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6682005Z ok (2.275s)
2023-01-11T21:18:21.6682011Z 
2023-01-11T21:18:21.6682214Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6682293Z Ran 1 test in 2.275s
2023-01-11T21:18:21.6682299Z 
2023-01-11T21:18:21.6682347Z OK
2023-01-11T21:18:21.6682366Z 
2023-01-11T21:18:21.6682439Z Generating XML reports...
2023-01-11T21:18:21.6682783Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210938.xml
2023-01-11T21:18:21.6683021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6683297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6683424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6683707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6683846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6683851Z 
2023-01-11T21:18:21.6683925Z Running tests...
2023-01-11T21:18:21.6684112Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6684812Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.258s)
2023-01-11T21:18:21.6684832Z 
2023-01-11T21:18:21.6685014Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6685090Z Ran 1 test in 0.258s
2023-01-11T21:18:21.6685095Z 
2023-01-11T21:18:21.6685166Z OK (skipped=1)
2023-01-11T21:18:21.6685171Z 
2023-01-11T21:18:21.6685296Z Generating XML reports...
2023-01-11T21:18:21.6685636Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210942.xml
2023-01-11T21:18:21.6685901Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6686180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6686306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6686574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6686712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6686717Z 
2023-01-11T21:18:21.6686790Z Running tests...
2023-01-11T21:18:21.6686990Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6687224Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19483
2023-01-11T21:18:21.6687387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19484
2023-01-11T21:18:21.6687544Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19485
2023-01-11T21:18:21.6687816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6687941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6688212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6688347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6688528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6688803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6688929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6689212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6689350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6689528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6689785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6689911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6690201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6690335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6690513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6690818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6691120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6691413Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6691582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6691734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6691929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6692106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6692322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6692524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6692820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6693119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6693409Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6693656Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6693884Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6694122Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6694374Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6694620Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6694869Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6695128Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6695383Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6695634Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6695878Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6696106Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6696349Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6696594Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6696839Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6697079Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6697334Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6697587Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6697837Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6698081Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6698309Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6698548Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6698789Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6699046Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6699291Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6699537Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6699791Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6700109Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6700349Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6700573Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6700813Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6701062Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6701303Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6701558Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6701937Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6702187Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6702441Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6702681Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6702906Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6703141Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6703389Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6703633Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6703889Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6704138Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6704399Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6704654Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6704893Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6705115Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6705354Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6705769Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6705777Z 
2023-01-11T21:18:21.6706028Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6706286Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6706540Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6706791Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6707033Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6707276Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6707516Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6707812Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6708089Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6708334Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6708589Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6708842Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6709095Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6709336Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6709746Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6709755Z 
2023-01-11T21:18:21.6709998Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6710253Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6710484Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6710728Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6711157Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6711165Z 
2023-01-11T21:18:21.6711406Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6711806Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6711812Z 
2023-01-11T21:18:21.6712222Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6712228Z 
2023-01-11T21:18:21.6712472Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6712728Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6712984Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6713242Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6713485Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6713709Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6713946Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6714190Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6714773Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6714823Z 
2023-01-11T21:18:21.6714827Z 
2023-01-11T21:18:21.6715283Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6715290Z 
2023-01-11T21:18:21.6715532Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6715768Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6716003Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6716251Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6716506Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6716750Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6716983Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6717407Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6717413Z 
2023-01-11T21:18:21.6717651Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6717889Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6718123Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6718370Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6718628Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6718873Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6719130Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6719375Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6719615Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6719861Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6720100Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6720337Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6720584Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6720829Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6721083Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6721330Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6721642Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6721885Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6722126Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6722365Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6722647Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6722927Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6723184Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6723595Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6723602Z 
2023-01-11T21:18:21.6724031Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6724038Z 
2023-01-11T21:18:21.6724279Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6724519Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6724756Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6724991Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6725246Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6725493Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6725747Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6725994Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6726249Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6726489Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6726727Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6726965Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6727197Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6727455Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6727702Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6727955Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6728202Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6728458Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6728695Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6728930Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6729165Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6729396Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6729638Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6729893Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6730168Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6730448Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6730703Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6730946Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6731187Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6731424Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6731826Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6731846Z 
2023-01-11T21:18:21.6732086Z STAGE:2023-01-11 21:09:46 19483:19483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6732342Z STAGE:2023-01-11 21:09:46 19485:19485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6732588Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6732842Z STAGE:2023-01-11 21:09:46 19484:19484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6732913Z ok (2.274s)
2023-01-11T21:18:21.6732919Z 
2023-01-11T21:18:21.6733121Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6733197Z Ran 1 test in 2.275s
2023-01-11T21:18:21.6733202Z 
2023-01-11T21:18:21.6733269Z OK
2023-01-11T21:18:21.6733274Z 
2023-01-11T21:18:21.6733358Z Generating XML reports...
2023-01-11T21:18:21.6733702Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210944.xml
2023-01-11T21:18:21.6733939Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6734223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6734347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6734631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6734768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6734774Z 
2023-01-11T21:18:21.6734849Z Running tests...
2023-01-11T21:18:21.6735051Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6735273Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19548
2023-01-11T21:18:21.6735434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19549
2023-01-11T21:18:21.6735591Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19550
2023-01-11T21:18:21.6735868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6735991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6736275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6736413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6736592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6736851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6737008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6737319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6737458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6737639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6737914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6738040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6738321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6738455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6738620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6738926Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6739228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6739521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6739690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6739855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6740021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6740196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.6740373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.6740537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.6740831Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6741123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6741374Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6741621Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6742151Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6742161Z 
2023-01-11T21:18:21.6742601Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6742607Z 
2023-01-11T21:18:21.6742850Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6743090Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6743340Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6743589Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6743836Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6744095Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6744398Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6744670Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6744920Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6745164Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6745424Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6745679Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6745922Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6746148Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6746398Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6746647Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6746904Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6747159Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6747400Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6747640Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6747888Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6748121Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6748378Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6748635Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6748874Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6749120Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6749368Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6749611Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6749871Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6750130Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6750359Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6750598Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6750846Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6751088Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6751339Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6751592Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6751831Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6752103Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6752418Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6752663Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6752904Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6753156Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6753397Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6753635Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6753880Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6754128Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6754388Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6754643Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6754872Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6755111Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6755357Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6755600Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6755858Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6756115Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6756358Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6756659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.6756900Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6757131Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6757386Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6757630Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6757886Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6758127Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6758366Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.6758610Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6758852Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.6759106Z STAGE:2023-01-11 21:09:50 19550:19550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6759344Z STAGE:2023-01-11 21:09:50 19549:19549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.6759417Z ok (2.279s)
2023-01-11T21:18:21.6759452Z 
2023-01-11T21:18:21.6759653Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6759730Z Ran 1 test in 2.280s
2023-01-11T21:18:21.6759735Z 
2023-01-11T21:18:21.6759797Z OK
2023-01-11T21:18:21.6759829Z 
2023-01-11T21:18:21.6759916Z Generating XML reports...
2023-01-11T21:18:21.6760262Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210949.xml
2023-01-11T21:18:21.6760497Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6760776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6760890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6761175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6761313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6761321Z 
2023-01-11T21:18:21.6761395Z Running tests...
2023-01-11T21:18:21.6761665Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6761911Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19609
2023-01-11T21:18:21.6762072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19610
2023-01-11T21:18:21.6762228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19611
2023-01-11T21:18:21.6762492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6762616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6762897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6763038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6763219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6763495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6763620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6763903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6764039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6764203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6764478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6764602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6764886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6765023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6765198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6765498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6765801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6766100Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6766253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6766418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6766632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6766754Z skip: CUDA is not available. (2.172s)
2023-01-11T21:18:21.6766760Z 
2023-01-11T21:18:21.6766959Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6767035Z Ran 1 test in 2.173s
2023-01-11T21:18:21.6767041Z 
2023-01-11T21:18:21.6767113Z OK (skipped=1)
2023-01-11T21:18:21.6767119Z 
2023-01-11T21:18:21.6767202Z Generating XML reports...
2023-01-11T21:18:21.6767532Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210953.xml
2023-01-11T21:18:21.6767767Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6768043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6768169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6768457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6768594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6768599Z 
2023-01-11T21:18:21.6768674Z Running tests...
2023-01-11T21:18:21.6768873Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6769568Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.264s)
2023-01-11T21:18:21.6769575Z 
2023-01-11T21:18:21.6769771Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6769837Z Ran 1 test in 0.264s
2023-01-11T21:18:21.6769856Z 
2023-01-11T21:18:21.6769916Z OK (skipped=1)
2023-01-11T21:18:21.6769921Z 
2023-01-11T21:18:21.6770007Z Generating XML reports...
2023-01-11T21:18:21.6770349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210957.xml
2023-01-11T21:18:21.6770586Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6770865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6770991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6771274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6771410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6771418Z 
2023-01-11T21:18:21.6771478Z Running tests...
2023-01-11T21:18:21.6771675Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6772058Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s)
2023-01-11T21:18:21.6772064Z 
2023-01-11T21:18:21.6772262Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6772339Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6772344Z 
2023-01-11T21:18:21.6772416Z OK (skipped=1)
2023-01-11T21:18:21.6772421Z 
2023-01-11T21:18:21.6772506Z Generating XML reports...
2023-01-11T21:18:21.6772845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210959.xml
2023-01-11T21:18:21.6773081Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6773344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6773504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6773809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6773946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6773951Z 
2023-01-11T21:18:21.6774027Z Running tests...
2023-01-11T21:18:21.6774224Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6774613Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s)
2023-01-11T21:18:21.6774620Z 
2023-01-11T21:18:21.6774815Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6774891Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6774899Z 
2023-01-11T21:18:21.6774957Z OK (skipped=1)
2023-01-11T21:18:21.6774976Z 
2023-01-11T21:18:21.6775048Z Generating XML reports...
2023-01-11T21:18:21.6775388Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211000.xml
2023-01-11T21:18:21.6775619Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6775893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6776017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6776295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6776431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6776437Z 
2023-01-11T21:18:21.6776510Z Running tests...
2023-01-11T21:18:21.6776694Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6776941Z   test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19692
2023-01-11T21:18:21.6777101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19693
2023-01-11T21:18:21.6777257Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19694
2023-01-11T21:18:21.6777534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6777659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6777942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6778080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6778247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6778522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6778649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6778930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6779066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6779246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6779551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6779822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6779945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6780254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6780416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6780589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6780889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6781185Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6781354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6781521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6782000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6782111Z skip: Need at least 2 CUDA devices (2.178s)
2023-01-11T21:18:21.6782118Z 
2023-01-11T21:18:21.6782304Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6782380Z Ran 1 test in 2.178s
2023-01-11T21:18:21.6782386Z 
2023-01-11T21:18:21.6782458Z OK (skipped=1)
2023-01-11T21:18:21.6782462Z 
2023-01-11T21:18:21.6782548Z Generating XML reports...
2023-01-11T21:18:21.6782891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211002.xml
2023-01-11T21:18:21.6783124Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6783405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6783530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6783817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6783939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6783946Z 
2023-01-11T21:18:21.6784020Z Running tests...
2023-01-11T21:18:21.6784217Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6784544Z   test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19745
2023-01-11T21:18:21.6784781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19746
2023-01-11T21:18:21.6784989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19747
2023-01-11T21:18:21.6785270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6785395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6785666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6785804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6785982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6786256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6786379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6786660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6786796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6786974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6787329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6787481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6787763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6787897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6788077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6788377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6788676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6788968Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6789137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6789299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6789451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6789554Z skip: Need at least 2 CUDA devices (2.181s)
2023-01-11T21:18:21.6789561Z 
2023-01-11T21:18:21.6789757Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6789832Z Ran 1 test in 2.181s
2023-01-11T21:18:21.6789838Z 
2023-01-11T21:18:21.6789911Z OK (skipped=1)
2023-01-11T21:18:21.6789916Z 
2023-01-11T21:18:21.6790001Z Generating XML reports...
2023-01-11T21:18:21.6790345Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211006.xml
2023-01-11T21:18:21.6790580Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6790855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6790967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6791245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6791378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6791383Z 
2023-01-11T21:18:21.6791455Z Running tests...
2023-01-11T21:18:21.6791650Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6791909Z   test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19798
2023-01-11T21:18:21.6792065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19799
2023-01-11T21:18:21.6792221Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19800
2023-01-11T21:18:21.6792481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6792604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6792884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6793019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6793194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6793466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6793589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6793901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6794034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6794227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6794500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6794622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6794900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6795049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6795289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6795597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6795898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6796192Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6796345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6796507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6796667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6796771Z skip: Need at least 2 CUDA devices (2.174s)
2023-01-11T21:18:21.6796778Z 
2023-01-11T21:18:21.6796977Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6797053Z Ran 1 test in 2.174s
2023-01-11T21:18:21.6797062Z 
2023-01-11T21:18:21.6797135Z OK (skipped=1)
2023-01-11T21:18:21.6797140Z 
2023-01-11T21:18:21.6797224Z Generating XML reports...
2023-01-11T21:18:21.6797555Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211010.xml
2023-01-11T21:18:21.6797790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6798064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6798186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6798463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6798597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6798603Z 
2023-01-11T21:18:21.6798675Z Running tests...
2023-01-11T21:18:21.6798873Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6799114Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19851
2023-01-11T21:18:21.6799257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19852
2023-01-11T21:18:21.6799412Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19853
2023-01-11T21:18:21.6799682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6799805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6800084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6800217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6800395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6800704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6800855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6801127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6801261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6801438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6801771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6801898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6802178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6802315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6802493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6802783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6803081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6803375Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6803541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6803706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6803867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6803977Z skip: Need at least 2 CUDA devices (2.184s)
2023-01-11T21:18:21.6803982Z 
2023-01-11T21:18:21.6804185Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6804261Z Ran 1 test in 2.185s
2023-01-11T21:18:21.6804266Z 
2023-01-11T21:18:21.6804324Z OK (skipped=1)
2023-01-11T21:18:21.6804328Z 
2023-01-11T21:18:21.6804415Z Generating XML reports...
2023-01-11T21:18:21.6804758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211014.xml
2023-01-11T21:18:21.6804991Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6805265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6805388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6805666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6805800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6805807Z 
2023-01-11T21:18:21.6805879Z Running tests...
2023-01-11T21:18:21.6806066Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6806308Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19904
2023-01-11T21:18:21.6806464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19905
2023-01-11T21:18:21.6806617Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19906
2023-01-11T21:18:21.6806892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6807014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6807337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6807501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6807666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6807964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6808234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6808354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6808633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6808767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6808945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6809218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6809339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6809605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6809740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6809915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6810215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6810511Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6810676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6810840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6811005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6811107Z skip: Need at least 2 CUDA devices (2.172s)
2023-01-11T21:18:21.6811112Z 
2023-01-11T21:18:21.6811295Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6811369Z Ran 1 test in 2.173s
2023-01-11T21:18:21.6811374Z 
2023-01-11T21:18:21.6811444Z OK (skipped=1)
2023-01-11T21:18:21.6811449Z 
2023-01-11T21:18:21.6811532Z Generating XML reports...
2023-01-11T21:18:21.6811875Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211018.xml
2023-01-11T21:18:21.6812105Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6812380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6812503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6812783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6812905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6812910Z 
2023-01-11T21:18:21.6812982Z Running tests...
2023-01-11T21:18:21.6813177Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6813876Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.253s)
2023-01-11T21:18:21.6813928Z 
2023-01-11T21:18:21.6814125Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6814228Z Ran 1 test in 0.253s
2023-01-11T21:18:21.6814233Z 
2023-01-11T21:18:21.6814304Z OK (skipped=1)
2023-01-11T21:18:21.6814309Z 
2023-01-11T21:18:21.6814393Z Generating XML reports...
2023-01-11T21:18:21.6814730Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211022.xml
2023-01-11T21:18:21.6814947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6815219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6815340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6815623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6815759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6815764Z 
2023-01-11T21:18:21.6815837Z Running tests...
2023-01-11T21:18:21.6816032Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6816744Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.260s)
2023-01-11T21:18:21.6816751Z 
2023-01-11T21:18:21.6816946Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6817020Z Ran 1 test in 0.260s
2023-01-11T21:18:21.6817026Z 
2023-01-11T21:18:21.6817085Z OK (skipped=1)
2023-01-11T21:18:21.6817105Z 
2023-01-11T21:18:21.6817176Z Generating XML reports...
2023-01-11T21:18:21.6817519Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211024.xml
2023-01-11T21:18:21.6817751Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6818026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6818149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6818428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6818563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6818568Z 
2023-01-11T21:18:21.6818640Z Running tests...
2023-01-11T21:18:21.6818825Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6819174Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.003s)
2023-01-11T21:18:21.6819180Z 
2023-01-11T21:18:21.6819373Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6819448Z Ran 1 test in 0.003s
2023-01-11T21:18:21.6819454Z 
2023-01-11T21:18:21.6819524Z OK (skipped=1)
2023-01-11T21:18:21.6819528Z 
2023-01-11T21:18:21.6819612Z Generating XML reports...
2023-01-11T21:18:21.6819949Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211026.xml
2023-01-11T21:18:21.6820178Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6820451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6820560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6820868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6821031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6821037Z 
2023-01-11T21:18:21.6821110Z Running tests...
2023-01-11T21:18:21.6821306Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6821573Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19987
2023-01-11T21:18:21.6821833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19988
2023-01-11T21:18:21.6821990Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19989
2023-01-11T21:18:21.6822261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6822374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6822659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6822795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6822975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6823247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6823369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6823650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6823788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6823952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6824225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6824349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6824629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6824764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6824941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6825242Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6825539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6825828Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6825985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6826148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6826310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6826414Z skip: Need at least 2 CUDA devices (2.180s)
2023-01-11T21:18:21.6826420Z 
2023-01-11T21:18:21.6826618Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6826692Z Ran 1 test in 2.181s
2023-01-11T21:18:21.6826698Z 
2023-01-11T21:18:21.6826769Z OK (skipped=1)
2023-01-11T21:18:21.6826774Z 
2023-01-11T21:18:21.6826857Z Generating XML reports...
2023-01-11T21:18:21.6827196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211028.xml
2023-01-11T21:18:21.6827473Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6827781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6827907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6828190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6828328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6828333Z 
2023-01-11T21:18:21.6828408Z Running tests...
2023-01-11T21:18:21.6828607Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6828846Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20040
2023-01-11T21:18:21.6829006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20041
2023-01-11T21:18:21.6829152Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20042
2023-01-11T21:18:21.6829425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6829550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6829831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6829969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6830147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6830421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6830545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6830818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6830958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6831135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6831408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6831532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6831812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6831949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6832125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6832430Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6832722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6833022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6833189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6833349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6833512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6833617Z skip: Need at least 3 CUDA devices (2.277s)
2023-01-11T21:18:21.6833623Z 
2023-01-11T21:18:21.6833825Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6833900Z Ran 1 test in 2.277s
2023-01-11T21:18:21.6833937Z 
2023-01-11T21:18:21.6834009Z OK (skipped=1)
2023-01-11T21:18:21.6834014Z 
2023-01-11T21:18:21.6834085Z Generating XML reports...
2023-01-11T21:18:21.6834454Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211032.xml
2023-01-11T21:18:21.6834687Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6834963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6835087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6835368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6835504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6835509Z 
2023-01-11T21:18:21.6835583Z Running tests...
2023-01-11T21:18:21.6835772Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6836126Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.004s)
2023-01-11T21:18:21.6836132Z 
2023-01-11T21:18:21.6836328Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6836405Z Ran 1 test in 0.004s
2023-01-11T21:18:21.6836410Z 
2023-01-11T21:18:21.6836482Z OK (skipped=1)
2023-01-11T21:18:21.6836488Z 
2023-01-11T21:18:21.6836572Z Generating XML reports...
2023-01-11T21:18:21.6836945Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211036.xml
2023-01-11T21:18:21.6837291Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6837620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6837733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6838019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6838154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6838160Z 
2023-01-11T21:18:21.6838232Z Running tests...
2023-01-11T21:18:21.6838427Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6838769Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.003s)
2023-01-11T21:18:21.6838774Z 
2023-01-11T21:18:21.6838968Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6839044Z Ran 1 test in 0.003s
2023-01-11T21:18:21.6839050Z 
2023-01-11T21:18:21.6839119Z OK (skipped=1)
2023-01-11T21:18:21.6839124Z 
2023-01-11T21:18:21.6839198Z Generating XML reports...
2023-01-11T21:18:21.6839546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211038.xml
2023-01-11T21:18:21.6839776Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6840050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6840173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6840451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6840584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6840589Z 
2023-01-11T21:18:21.6840662Z Running tests...
2023-01-11T21:18:21.6840856Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6841075Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20113
2023-01-11T21:18:21.6841310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20114
2023-01-11T21:18:21.6841467Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20115
2023-01-11T21:18:21.6841818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6841946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6842230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6842367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6842546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6842806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6842936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6843221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6843357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6843538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6843838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6844111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6844235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6844512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6844635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6844811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6845110Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6845409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6845575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6845738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6845901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6846598Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6847284Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6847953Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6848926Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.)
2023-01-11T21:18:21.6849099Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2023-01-11T21:18:21.6849272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.6850157Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.)
2023-01-11T21:18:21.6850329Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2023-01-11T21:18:21.6850498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.6851387Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.)
2023-01-11T21:18:21.6851556Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2023-01-11T21:18:21.6851711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.6852394Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6853083Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6853771Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6854442Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6855174Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6855846Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6856520Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6857210Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6857876Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6858547Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6859221Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6859888Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6860559Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6861220Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6862081Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:18:21.6862187Z ok (2.179s)
2023-01-11T21:18:21.6862194Z 
2023-01-11T21:18:21.6862383Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6862461Z Ran 1 test in 2.179s
2023-01-11T21:18:21.6862468Z 
2023-01-11T21:18:21.6862530Z OK
2023-01-11T21:18:21.6862535Z 
2023-01-11T21:18:21.6862623Z Generating XML reports...
2023-01-11T21:18:21.6862968Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211040.xml
2023-01-11T21:18:21.6863206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6863486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6863613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6863897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6864021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6864026Z 
2023-01-11T21:18:21.6864099Z Running tests...
2023-01-11T21:18:21.6864296Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6864603Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.004s)
2023-01-11T21:18:21.6864608Z 
2023-01-11T21:18:21.6864799Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6864875Z Ran 1 test in 0.004s
2023-01-11T21:18:21.6864880Z 
2023-01-11T21:18:21.6864950Z OK (skipped=1)
2023-01-11T21:18:21.6864955Z 
2023-01-11T21:18:21.6865040Z Generating XML reports...
2023-01-11T21:18:21.6865365Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211044.xml
2023-01-11T21:18:21.6865598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6865871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6865993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6866275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6866410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6866416Z 
2023-01-11T21:18:21.6866487Z Running tests...
2023-01-11T21:18:21.6866684Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6867007Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.003s)
2023-01-11T21:18:21.6867015Z 
2023-01-11T21:18:21.6867209Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6867272Z Ran 1 test in 0.003s
2023-01-11T21:18:21.6867277Z 
2023-01-11T21:18:21.6867347Z OK (skipped=1)
2023-01-11T21:18:21.6867352Z 
2023-01-11T21:18:21.6867434Z Generating XML reports...
2023-01-11T21:18:21.6867768Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211045.xml
2023-01-11T21:18:21.6867999Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6868272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6868399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6868675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6868827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6868871Z 
2023-01-11T21:18:21.6868932Z Running tests...
2023-01-11T21:18:21.6869124Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6869815Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.262s)
2023-01-11T21:18:21.6869821Z 
2023-01-11T21:18:21.6870016Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6870090Z Ran 1 test in 0.263s
2023-01-11T21:18:21.6870096Z 
2023-01-11T21:18:21.6870169Z OK (skipped=1)
2023-01-11T21:18:21.6870174Z 
2023-01-11T21:18:21.6870256Z Generating XML reports...
2023-01-11T21:18:21.6870593Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211047.xml
2023-01-11T21:18:21.6870823Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6871081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6871202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6871477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6871612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6871617Z 
2023-01-11T21:18:21.6871690Z Running tests...
2023-01-11T21:18:21.6871883Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6872581Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.262s)
2023-01-11T21:18:21.6872588Z 
2023-01-11T21:18:21.6872785Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6872863Z Ran 1 test in 0.262s
2023-01-11T21:18:21.6872868Z 
2023-01-11T21:18:21.6872938Z OK (skipped=1)
2023-01-11T21:18:21.6872943Z 
2023-01-11T21:18:21.6873014Z Generating XML reports...
2023-01-11T21:18:21.6873354Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211049.xml
2023-01-11T21:18:21.6873583Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6873857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6873981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6874263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6874397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6874403Z 
2023-01-11T21:18:21.6874474Z Running tests...
2023-01-11T21:18:21.6874657Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6874911Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20206
2023-01-11T21:18:21.6875067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20207
2023-01-11T21:18:21.6875221Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20208
2023-01-11T21:18:21.6875525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6875674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6875955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6876089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6876268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6876528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6876648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6876928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6877066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6877246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6877517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6877640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6877918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6878049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6878212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6878514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6878821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6879116Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6879280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6879443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6879606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6879709Z skip: Need at least 3 CUDA devices (2.180s)
2023-01-11T21:18:21.6879714Z 
2023-01-11T21:18:21.6879910Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6879972Z Ran 1 test in 2.180s
2023-01-11T21:18:21.6879977Z 
2023-01-11T21:18:21.6880048Z OK (skipped=1)
2023-01-11T21:18:21.6880055Z 
2023-01-11T21:18:21.6880138Z Generating XML reports...
2023-01-11T21:18:21.6880482Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211051.xml
2023-01-11T21:18:21.6880713Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6880983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6881106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6881388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6881510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6881528Z 
2023-01-11T21:18:21.6881656Z Running tests...
2023-01-11T21:18:21.6881861Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6882139Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20259
2023-01-11T21:18:21.6882326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20260
2023-01-11T21:18:21.6882482Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20261
2023-01-11T21:18:21.6882758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6882881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6883160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6883283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6883462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6883736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6883860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6884142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6884277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6884457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6884726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6884848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6885112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6885246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6885423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6885723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6886019Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6886309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6886475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6886636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6886798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6886892Z skip: Need at least 3 CUDA devices (2.178s)
2023-01-11T21:18:21.6886898Z 
2023-01-11T21:18:21.6887095Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6887172Z Ran 1 test in 2.178s
2023-01-11T21:18:21.6887178Z 
2023-01-11T21:18:21.6887249Z OK (skipped=1)
2023-01-11T21:18:21.6887254Z 
2023-01-11T21:18:21.6887338Z Generating XML reports...
2023-01-11T21:18:21.6887680Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211055.xml
2023-01-11T21:18:21.6887911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6888185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6888295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6888571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6888735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6888740Z 
2023-01-11T21:18:21.6888813Z Running tests...
2023-01-11T21:18:21.6889040Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6889736Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2023-01-11T21:18:21.6889742Z 
2023-01-11T21:18:21.6889938Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6890012Z Ran 1 test in 0.255s
2023-01-11T21:18:21.6890018Z 
2023-01-11T21:18:21.6890089Z OK (skipped=1)
2023-01-11T21:18:21.6890094Z 
2023-01-11T21:18:21.6890179Z Generating XML reports...
2023-01-11T21:18:21.6890505Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211059.xml
2023-01-11T21:18:21.6890738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6891011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6891138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6891417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6891552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6891557Z 
2023-01-11T21:18:21.6891631Z Running tests...
2023-01-11T21:18:21.6891826Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6892063Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20322
2023-01-11T21:18:21.6892213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20323
2023-01-11T21:18:21.6892367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20324
2023-01-11T21:18:21.6892639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6892762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6893041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6893176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6893354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6893624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6893735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6894015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6894148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6894329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6894598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6894719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6895000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6895134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6895339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6895653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6895958Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6896254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6896419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6896587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6896752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6896857Z skip: Need at least 3 CUDA devices (2.275s)
2023-01-11T21:18:21.6896865Z 
2023-01-11T21:18:21.6897062Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6897138Z Ran 1 test in 2.276s
2023-01-11T21:18:21.6897146Z 
2023-01-11T21:18:21.6897204Z OK (skipped=1)
2023-01-11T21:18:21.6897210Z 
2023-01-11T21:18:21.6897298Z Generating XML reports...
2023-01-11T21:18:21.6897642Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211101.xml
2023-01-11T21:18:21.6897874Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6898149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6898272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6898555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6898693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6898698Z 
2023-01-11T21:18:21.6898772Z Running tests...
2023-01-11T21:18:21.6898960Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6899231Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20375
2023-01-11T21:18:21.6899391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20376
2023-01-11T21:18:21.6899548Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20377
2023-01-11T21:18:21.6899820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6899944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6900224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6900362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6900528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6900800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6900925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6901209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6901343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6901519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6902007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6902190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6902519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6902644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6902821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6903120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6903421Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6903720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6903886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6904051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6904216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6904323Z skip: Need at least 2 CUDA devices (2.277s)
2023-01-11T21:18:21.6904329Z 
2023-01-11T21:18:21.6904516Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6904595Z Ran 1 test in 2.278s
2023-01-11T21:18:21.6904600Z 
2023-01-11T21:18:21.6904672Z OK (skipped=1)
2023-01-11T21:18:21.6904677Z 
2023-01-11T21:18:21.6904762Z Generating XML reports...
2023-01-11T21:18:21.6905107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211105.xml
2023-01-11T21:18:21.6905342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6905624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6905750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6906020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6906159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6906165Z 
2023-01-11T21:18:21.6906240Z Running tests...
2023-01-11T21:18:21.6906435Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6906704Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20428
2023-01-11T21:18:21.6906863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20429
2023-01-11T21:18:21.6907021Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20430
2023-01-11T21:18:21.6907295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6907419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6907684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6907821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6907997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6908266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6908388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6908667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6908858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6909061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6909362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6909617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6909739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6910014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6910149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6910326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6910624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6910925Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6911088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6911252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6911397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6911500Z skip: Need at least 2 CUDA devices (2.177s)
2023-01-11T21:18:21.6911505Z 
2023-01-11T21:18:21.6911703Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6911778Z Ran 1 test in 2.177s
2023-01-11T21:18:21.6911784Z 
2023-01-11T21:18:21.6911854Z OK (skipped=1)
2023-01-11T21:18:21.6911859Z 
2023-01-11T21:18:21.6911945Z Generating XML reports...
2023-01-11T21:18:21.6912285Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211109.xml
2023-01-11T21:18:21.6912517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6912775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6912898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6913176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6913310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6913315Z 
2023-01-11T21:18:21.6913386Z Running tests...
2023-01-11T21:18:21.6913580Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6913898Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20481
2023-01-11T21:18:21.6914060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20482
2023-01-11T21:18:21.6914213Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20483
2023-01-11T21:18:21.6914472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6914595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6914875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6915010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6915186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6915492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6915642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6915924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6916058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6916223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6916492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6916615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6916896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6917032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6917208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6917505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6917799Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6918095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6918246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6918408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6918569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6918675Z skip: Need at least 2 CUDA devices (2.180s)
2023-01-11T21:18:21.6918681Z 
2023-01-11T21:18:21.6918878Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6918952Z Ran 1 test in 2.180s
2023-01-11T21:18:21.6918958Z 
2023-01-11T21:18:21.6919028Z OK (skipped=1)
2023-01-11T21:18:21.6919033Z 
2023-01-11T21:18:21.6919118Z Generating XML reports...
2023-01-11T21:18:21.6919446Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211113.xml
2023-01-11T21:18:21.6919677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6919948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6920073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6920352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6920488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6920495Z 
2023-01-11T21:18:21.6920569Z Running tests...
2023-01-11T21:18:21.6920766Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6921084Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20534
2023-01-11T21:18:21.6921230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20535
2023-01-11T21:18:21.6921384Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20536
2023-01-11T21:18:21.6921747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6921877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6922210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6922374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6922553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6922826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6922949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6923216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6923351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6923530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6923834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6924110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6924232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6924513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6924648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6924825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6925110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6925408Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6925574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6925739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6925902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6926004Z skip: Need at least 2 CUDA devices (2.171s)
2023-01-11T21:18:21.6926009Z 
2023-01-11T21:18:21.6926204Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6926279Z Ran 1 test in 2.171s
2023-01-11T21:18:21.6926284Z 
2023-01-11T21:18:21.6926343Z OK (skipped=1)
2023-01-11T21:18:21.6926360Z 
2023-01-11T21:18:21.6926432Z Generating XML reports...
2023-01-11T21:18:21.6926774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211117.xml
2023-01-11T21:18:21.6927009Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6927281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6927404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6927683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6927817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6927822Z 
2023-01-11T21:18:21.6927895Z Running tests...
2023-01-11T21:18:21.6928077Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6928393Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20587
2023-01-11T21:18:21.6928582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20588
2023-01-11T21:18:21.6928736Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20589
2023-01-11T21:18:21.6929039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6929162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6929445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6929577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6929757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6930016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6930136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6930414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6930550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6930726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6931022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6931292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6931412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6931689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6931810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6931987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6932284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6932579Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6932742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6932905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6933067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6933170Z skip: Need at least 2 CUDA devices (2.172s)
2023-01-11T21:18:21.6933175Z 
2023-01-11T21:18:21.6933372Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6933437Z Ran 1 test in 2.173s
2023-01-11T21:18:21.6933442Z 
2023-01-11T21:18:21.6933511Z OK (skipped=1)
2023-01-11T21:18:21.6933516Z 
2023-01-11T21:18:21.6933600Z Generating XML reports...
2023-01-11T21:18:21.6933942Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211121.xml
2023-01-11T21:18:21.6934174Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6934445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6934566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6934847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6934970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6934986Z 
2023-01-11T21:18:21.6935083Z Running tests...
2023-01-11T21:18:21.6935279Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6935621Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20640
2023-01-11T21:18:21.6935779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20641
2023-01-11T21:18:21.6935934Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20642
2023-01-11T21:18:21.6936204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6936327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6936604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6936729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6936905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6937177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6937298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6937577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6937711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6937887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6938156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6938276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6938544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6938678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6938855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6939154Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6939450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6939738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6939904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6940067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6940230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6940323Z skip: Need at least 2 CUDA devices (2.173s)
2023-01-11T21:18:21.6940329Z 
2023-01-11T21:18:21.6940526Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6940601Z Ran 1 test in 2.173s
2023-01-11T21:18:21.6940606Z 
2023-01-11T21:18:21.6940677Z OK (skipped=1)
2023-01-11T21:18:21.6940682Z 
2023-01-11T21:18:21.6940766Z Generating XML reports...
2023-01-11T21:18:21.6941106Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211125.xml
2023-01-11T21:18:21.6941337Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6941605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6941864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6942201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6942338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6942344Z 
2023-01-11T21:18:21.6942418Z Running tests...
2023-01-11T21:18:21.6942618Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6942934Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20693
2023-01-11T21:18:21.6943092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20694
2023-01-11T21:18:21.6943247Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20695
2023-01-11T21:18:21.6943526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6943637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6943923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6944057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6944235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6944509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6944635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6944918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6945052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6945231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6945492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6945617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6945896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6946031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6946207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6946506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6946803Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6947097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6947260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6947410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6947570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6947675Z skip: Need at least 2 CUDA devices (2.174s)
2023-01-11T21:18:21.6947680Z 
2023-01-11T21:18:21.6947877Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6947951Z Ran 1 test in 2.175s
2023-01-11T21:18:21.6947957Z 
2023-01-11T21:18:21.6948028Z OK (skipped=1)
2023-01-11T21:18:21.6948033Z 
2023-01-11T21:18:21.6948118Z Generating XML reports...
2023-01-11T21:18:21.6948502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211129.xml
2023-01-11T21:18:21.6948750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6949027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6949150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6949430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6949565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6949570Z 
2023-01-11T21:18:21.6949643Z Running tests...
2023-01-11T21:18:21.6949839Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6950154Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20746
2023-01-11T21:18:21.6950313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20747
2023-01-11T21:18:21.6950454Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20748
2023-01-11T21:18:21.6950724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6950847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6951127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6951262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6951438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6951712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6951836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6952121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6952244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6952423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6952691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6952816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6953095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6953233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6953410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6953713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6954012Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6954289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6954456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6954621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6954784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6954935Z skip: Need at least 2 CUDA devices (2.178s)
2023-01-11T21:18:21.6954941Z 
2023-01-11T21:18:21.6955139Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6955242Z Ran 1 test in 2.179s
2023-01-11T21:18:21.6955249Z 
2023-01-11T21:18:21.6955323Z OK (skipped=1)
2023-01-11T21:18:21.6955328Z 
2023-01-11T21:18:21.6955401Z Generating XML reports...
2023-01-11T21:18:21.6955742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211133.xml
2023-01-11T21:18:21.6955975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6956251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6956375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6956658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6956799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6956804Z 
2023-01-11T21:18:21.6956877Z Running tests...
2023-01-11T21:18:21.6957080Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6957385Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20799
2023-01-11T21:18:21.6957543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20800
2023-01-11T21:18:21.6957696Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20801
2023-01-11T21:18:21.6957970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6958093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6958376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6958516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6958694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6958967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6959076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6959360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6959494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6959673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6959945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6960070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6960352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6960486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6960664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6960950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6961249Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6961540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6961816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6962042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6962206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6962311Z skip: Need at least 2 CUDA devices (2.180s)
2023-01-11T21:18:21.6962316Z 
2023-01-11T21:18:21.6962518Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6962595Z Ran 1 test in 2.181s
2023-01-11T21:18:21.6962600Z 
2023-01-11T21:18:21.6962662Z OK (skipped=1)
2023-01-11T21:18:21.6962667Z 
2023-01-11T21:18:21.6962751Z Generating XML reports...
2023-01-11T21:18:21.6963094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211137.xml
2023-01-11T21:18:21.6963327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6963605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6963729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6964009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6964146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6964151Z 
2023-01-11T21:18:21.6964210Z Running tests...
2023-01-11T21:18:21.6964407Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6964722Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20852
2023-01-11T21:18:21.6964879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20853
2023-01-11T21:18:21.6965036Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20854
2023-01-11T21:18:21.6965311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6965433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6965710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6965844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6966009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6966279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6966399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6966680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6966817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6966993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6967263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6967386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6967662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6967782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6967956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6968251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6968600Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6968893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6969059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6969221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6969381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6969484Z skip: Need at least 2 CUDA devices (2.178s)
2023-01-11T21:18:21.6969490Z 
2023-01-11T21:18:21.6969674Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6969748Z Ran 1 test in 2.179s
2023-01-11T21:18:21.6969755Z 
2023-01-11T21:18:21.6969827Z OK (skipped=1)
2023-01-11T21:18:21.6969832Z 
2023-01-11T21:18:21.6969915Z Generating XML reports...
2023-01-11T21:18:21.6970258Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211141.xml
2023-01-11T21:18:21.6970492Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6970761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6970883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6971148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6971285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6971290Z 
2023-01-11T21:18:21.6971363Z Running tests...
2023-01-11T21:18:21.6971560Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6971832Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20905
2023-01-11T21:18:21.6971989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20906
2023-01-11T21:18:21.6972142Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20907
2023-01-11T21:18:21.6972413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6972535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6972804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6972939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6973118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6973390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6973512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6973794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6973927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6974103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6974372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6974483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6974761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6974926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6975131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6975433Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6975732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6976023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6976188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6976348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6976498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6976601Z skip: Need at least 2 CUDA devices (2.188s)
2023-01-11T21:18:21.6976607Z 
2023-01-11T21:18:21.6976807Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6976882Z Ran 1 test in 2.188s
2023-01-11T21:18:21.6976887Z 
2023-01-11T21:18:21.6976959Z OK (skipped=1)
2023-01-11T21:18:21.6976964Z 
2023-01-11T21:18:21.6977047Z Generating XML reports...
2023-01-11T21:18:21.6977391Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211145.xml
2023-01-11T21:18:21.6977624Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6977888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6978010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6978293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6978428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6978433Z 
2023-01-11T21:18:21.6978504Z Running tests...
2023-01-11T21:18:21.6978701Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6978975Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20958
2023-01-11T21:18:21.6979130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20959
2023-01-11T21:18:21.6979283Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20960
2023-01-11T21:18:21.6979540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6979666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6979947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6980081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6980257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.6980528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6980650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6980929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6981060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6981224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.6981526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6981787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6982073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6982208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6982385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.6982682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6982985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6983265Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.6983433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.6995279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.6995492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.6995606Z skip: Need at least 2 CUDA devices (2.175s)
2023-01-11T21:18:21.6995613Z 
2023-01-11T21:18:21.6995853Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6995930Z Ran 1 test in 2.175s
2023-01-11T21:18:21.6995936Z 
2023-01-11T21:18:21.6996011Z OK (skipped=1)
2023-01-11T21:18:21.6996017Z 
2023-01-11T21:18:21.6996106Z Generating XML reports...
2023-01-11T21:18:21.6996451Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211149.xml
2023-01-11T21:18:21.6996694Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6996979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6997105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6997391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6997528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.6997535Z 
2023-01-11T21:18:21.6997612Z Running tests...
2023-01-11T21:18:21.6997810Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6998137Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s)
2023-01-11T21:18:21.6998143Z 
2023-01-11T21:18:21.6998325Z ----------------------------------------------------------------------
2023-01-11T21:18:21.6998407Z Ran 1 test in 0.002s
2023-01-11T21:18:21.6998412Z 
2023-01-11T21:18:21.6998483Z OK (skipped=1)
2023-01-11T21:18:21.6998489Z 
2023-01-11T21:18:21.6998576Z Generating XML reports...
2023-01-11T21:18:21.6998919Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211153.xml
2023-01-11T21:18:21.6999153Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.6999433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.6999558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.6999841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.6999966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7000119Z 
2023-01-11T21:18:21.7000196Z Running tests...
2023-01-11T21:18:21.7000394Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7000673Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21021
2023-01-11T21:18:21.7000836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21022
2023-01-11T21:18:21.7000992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21023
2023-01-11T21:18:21.7001271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7001396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7001753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7001892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7002075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7002352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7002476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7002759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7002894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7003073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7003347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7003457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7003744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7003883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7004060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7004365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7004666Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7004962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7005127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7005294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7005447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7005553Z skip: Need at least 2 CUDA devices (2.173s)
2023-01-11T21:18:21.7005559Z 
2023-01-11T21:18:21.7005757Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7005833Z Ran 1 test in 2.173s
2023-01-11T21:18:21.7005838Z 
2023-01-11T21:18:21.7005913Z OK (skipped=1)
2023-01-11T21:18:21.7005918Z 
2023-01-11T21:18:21.7006005Z Generating XML reports...
2023-01-11T21:18:21.7006351Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211155.xml
2023-01-11T21:18:21.7006586Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7006849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7007012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7007295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7007461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7007467Z 
2023-01-11T21:18:21.7007543Z Running tests...
2023-01-11T21:18:21.7007739Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7007983Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21074
2023-01-11T21:18:21.7008140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21075
2023-01-11T21:18:21.7008297Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21076
2023-01-11T21:18:21.7008561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7008689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7008974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7009110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7009287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7009559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7009685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7009969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7010105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7010270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7010545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7010668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7010949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7011084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7011260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7011562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7011864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7012160Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7012315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7012480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7012642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7012747Z skip: Need at least 2 CUDA devices (2.177s)
2023-01-11T21:18:21.7012752Z 
2023-01-11T21:18:21.7012953Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7013029Z Ran 1 test in 2.177s
2023-01-11T21:18:21.7013035Z 
2023-01-11T21:18:21.7013108Z OK (skipped=1)
2023-01-11T21:18:21.7013113Z 
2023-01-11T21:18:21.7013199Z Generating XML reports...
2023-01-11T21:18:21.7013527Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211159.xml
2023-01-11T21:18:21.7013816Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7014120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7014248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7014529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7014667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7014673Z 
2023-01-11T21:18:21.7014749Z Running tests...
2023-01-11T21:18:21.7014947Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7015181Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21127
2023-01-11T21:18:21.7015326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21128
2023-01-11T21:18:21.7015480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21129
2023-01-11T21:18:21.7015756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7015883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7016163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7016299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7016477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7016747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7016858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7017142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7017281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7017460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7017737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7017862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7018140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7018276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7018455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7018748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7019056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7019352Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7019520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7019686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7019848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7020019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.7020188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.7020385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:18:21.7020442Z ok (2.283s)
2023-01-11T21:18:21.7020448Z 
2023-01-11T21:18:21.7020677Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7020756Z Ran 1 test in 2.284s
2023-01-11T21:18:21.7020762Z 
2023-01-11T21:18:21.7020824Z OK
2023-01-11T21:18:21.7020830Z 
2023-01-11T21:18:21.7020915Z Generating XML reports...
2023-01-11T21:18:21.7021259Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211203.xml
2023-01-11T21:18:21.7021493Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7021935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7022049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7022337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7022477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7022482Z 
2023-01-11T21:18:21.7022557Z Running tests...
2023-01-11T21:18:21.7022756Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7022992Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21189
2023-01-11T21:18:21.7023152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21190
2023-01-11T21:18:21.7023310Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21191
2023-01-11T21:18:21.7023580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7023693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7023978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7024115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7024294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7024597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7024868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7024993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7025275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7025411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7025577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7025853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7025978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7026260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7026395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7026573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7026872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7027170Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7027370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7027568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7027735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7027832Z skip: CUDA is not available. (2.177s)
2023-01-11T21:18:21.7027838Z 
2023-01-11T21:18:21.7028037Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7028115Z Ran 1 test in 2.178s
2023-01-11T21:18:21.7028121Z 
2023-01-11T21:18:21.7028195Z OK (skipped=1)
2023-01-11T21:18:21.7028201Z 
2023-01-11T21:18:21.7028287Z Generating XML reports...
2023-01-11T21:18:21.7028633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211207.xml
2023-01-11T21:18:21.7028854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7029133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7029261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7029546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7029684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7029690Z 
2023-01-11T21:18:21.7029764Z Running tests...
2023-01-11T21:18:21.7029963Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7030319Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s)
2023-01-11T21:18:21.7030325Z 
2023-01-11T21:18:21.7030521Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7030585Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7030591Z 
2023-01-11T21:18:21.7030664Z OK (skipped=1)
2023-01-11T21:18:21.7030669Z 
2023-01-11T21:18:21.7030758Z Generating XML reports...
2023-01-11T21:18:21.7031100Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211211.xml
2023-01-11T21:18:21.7031332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7031609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7031733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7032019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7032155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7032162Z 
2023-01-11T21:18:21.7032222Z Running tests...
2023-01-11T21:18:21.7032417Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7032761Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s)
2023-01-11T21:18:21.7032767Z 
2023-01-11T21:18:21.7032963Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7033039Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7033045Z 
2023-01-11T21:18:21.7033118Z OK (skipped=1)
2023-01-11T21:18:21.7033123Z 
2023-01-11T21:18:21.7033208Z Generating XML reports...
2023-01-11T21:18:21.7033550Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211213.xml
2023-01-11T21:18:21.7033783Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7034047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7034208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7034519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7034659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7034664Z 
2023-01-11T21:18:21.7034739Z Running tests...
2023-01-11T21:18:21.7034937Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7035318Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s)
2023-01-11T21:18:21.7035324Z 
2023-01-11T21:18:21.7035520Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7035583Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7035603Z 
2023-01-11T21:18:21.7035663Z OK (skipped=1)
2023-01-11T21:18:21.7035669Z 
2023-01-11T21:18:21.7035755Z Generating XML reports...
2023-01-11T21:18:21.7036095Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211215.xml
2023-01-11T21:18:21.7036329Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7036603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7036731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7037014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7037151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7037157Z 
2023-01-11T21:18:21.7037217Z Running tests...
2023-01-11T21:18:21.7037414Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7037771Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s)
2023-01-11T21:18:21.7037778Z 
2023-01-11T21:18:21.7037976Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7038053Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7038060Z 
2023-01-11T21:18:21.7038133Z OK (skipped=1)
2023-01-11T21:18:21.7038138Z 
2023-01-11T21:18:21.7038223Z Generating XML reports...
2023-01-11T21:18:21.7038565Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211216.xml
2023-01-11T21:18:21.7038797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7039058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7039184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7039467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7039606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7039612Z 
2023-01-11T21:18:21.7039686Z Running tests...
2023-01-11T21:18:21.7039884Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7040198Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s)
2023-01-11T21:18:21.7040203Z 
2023-01-11T21:18:21.7040403Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7040479Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7040484Z 
2023-01-11T21:18:21.7040542Z OK (skipped=1)
2023-01-11T21:18:21.7040547Z 
2023-01-11T21:18:21.7040633Z Generating XML reports...
2023-01-11T21:18:21.7040972Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211218.xml
2023-01-11T21:18:21.7041266Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7041539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7041734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7042025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7042162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7042168Z 
2023-01-11T21:18:21.7042241Z Running tests...
2023-01-11T21:18:21.7042424Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7042659Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21292
2023-01-11T21:18:21.7042820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21293
2023-01-11T21:18:21.7042981Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21294
2023-01-11T21:18:21.7043257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7043387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7043671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7043808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7043972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7044250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7044375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7044660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7044797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7044978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7045251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7045376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7045656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7045779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7045955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7046261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7046594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7046982Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7047152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7047318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7047480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7047587Z skip: Need at least 2 CUDA devices (2.172s)
2023-01-11T21:18:21.7047592Z 
2023-01-11T21:18:21.7047826Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7047903Z Ran 1 test in 2.172s
2023-01-11T21:18:21.7047909Z 
2023-01-11T21:18:21.7047981Z OK (skipped=1)
2023-01-11T21:18:21.7048015Z 
2023-01-11T21:18:21.7048102Z Generating XML reports...
2023-01-11T21:18:21.7048446Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211220.xml
2023-01-11T21:18:21.7048680Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7048957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7049082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7049351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7049489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7049497Z 
2023-01-11T21:18:21.7049571Z Running tests...
2023-01-11T21:18:21.7049793Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7050510Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.260s)
2023-01-11T21:18:21.7050516Z 
2023-01-11T21:18:21.7050710Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7050787Z Ran 1 test in 0.260s
2023-01-11T21:18:21.7050792Z 
2023-01-11T21:18:21.7050862Z OK (skipped=1)
2023-01-11T21:18:21.7050867Z 
2023-01-11T21:18:21.7050965Z Generating XML reports...
2023-01-11T21:18:21.7051308Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211224.xml
2023-01-11T21:18:21.7051531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7051808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7051939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7052220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7052356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7052362Z 
2023-01-11T21:18:21.7052439Z Running tests...
2023-01-11T21:18:21.7052636Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7052979Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s)
2023-01-11T21:18:21.7052988Z 
2023-01-11T21:18:21.7053185Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7053248Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7053269Z 
2023-01-11T21:18:21.7053327Z OK (skipped=1)
2023-01-11T21:18:21.7053332Z 
2023-01-11T21:18:21.7053418Z Generating XML reports...
2023-01-11T21:18:21.7053759Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211226.xml
2023-01-11T21:18:21.7053993Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7054270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7054393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7054677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7054843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7054849Z 
2023-01-11T21:18:21.7054911Z Running tests...
2023-01-11T21:18:21.7055139Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7055477Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s)
2023-01-11T21:18:21.7055483Z 
2023-01-11T21:18:21.7055676Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7055752Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7055758Z 
2023-01-11T21:18:21.7055829Z OK (skipped=1)
2023-01-11T21:18:21.7055835Z 
2023-01-11T21:18:21.7055920Z Generating XML reports...
2023-01-11T21:18:21.7056258Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211227.xml
2023-01-11T21:18:21.7056491Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7056754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7056880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7057163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7057297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7057303Z 
2023-01-11T21:18:21.7057376Z Running tests...
2023-01-11T21:18:21.7057570Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7057812Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21375
2023-01-11T21:18:21.7057971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21376
2023-01-11T21:18:21.7058118Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21377
2023-01-11T21:18:21.7058396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7058524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7058807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7058948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7059128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7059403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7059526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7059805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7059929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7060110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7060382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7060507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7060786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7060923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7061100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7061401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7061843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7062181Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7062350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7062518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7062682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7062788Z skip: Need at least 2 CUDA devices (2.176s)
2023-01-11T21:18:21.7062794Z 
2023-01-11T21:18:21.7062994Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7063070Z Ran 1 test in 2.176s
2023-01-11T21:18:21.7063075Z 
2023-01-11T21:18:21.7063150Z OK (skipped=1)
2023-01-11T21:18:21.7063155Z 
2023-01-11T21:18:21.7063240Z Generating XML reports...
2023-01-11T21:18:21.7063570Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211229.xml
2023-01-11T21:18:21.7063803Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7064078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7064202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7064486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7064621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7064626Z 
2023-01-11T21:18:21.7064700Z Running tests...
2023-01-11T21:18:21.7064898Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7065608Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.256s)
2023-01-11T21:18:21.7065615Z 
2023-01-11T21:18:21.7065811Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7065874Z Ran 1 test in 0.256s
2023-01-11T21:18:21.7065878Z 
2023-01-11T21:18:21.7065951Z OK (skipped=1)
2023-01-11T21:18:21.7065956Z 
2023-01-11T21:18:21.7066041Z Generating XML reports...
2023-01-11T21:18:21.7066381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211233.xml
2023-01-11T21:18:21.7066615Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7066894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7067022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7067305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7067428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7067447Z 
2023-01-11T21:18:21.7067508Z Running tests...
2023-01-11T21:18:21.7067706Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7068045Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s)
2023-01-11T21:18:21.7068051Z 
2023-01-11T21:18:21.7068246Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7068362Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7068368Z 
2023-01-11T21:18:21.7068439Z OK (skipped=1)
2023-01-11T21:18:21.7068444Z 
2023-01-11T21:18:21.7068531Z Generating XML reports...
2023-01-11T21:18:21.7068903Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211235.xml
2023-01-11T21:18:21.7069126Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7069407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7069531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7069813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7069948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7069954Z 
2023-01-11T21:18:21.7070027Z Running tests...
2023-01-11T21:18:21.7070228Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7070931Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.258s)
2023-01-11T21:18:21.7070938Z 
2023-01-11T21:18:21.7071135Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7071211Z Ran 1 test in 0.258s
2023-01-11T21:18:21.7071217Z 
2023-01-11T21:18:21.7071275Z OK (skipped=1)
2023-01-11T21:18:21.7071280Z 
2023-01-11T21:18:21.7071369Z Generating XML reports...
2023-01-11T21:18:21.7071707Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211237.xml
2023-01-11T21:18:21.7071948Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7072224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7072350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7072630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7072766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7072772Z 
2023-01-11T21:18:21.7072848Z Running tests...
2023-01-11T21:18:21.7073032Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7073273Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21458
2023-01-11T21:18:21.7073433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21459
2023-01-11T21:18:21.7073594Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21460
2023-01-11T21:18:21.7073873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7073999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7074283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7074423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7074588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7074891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7075165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7075325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7075635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7075773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7075949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7076224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7076349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7076616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7076753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7076933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7077241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7077538Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7077705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7077872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7078037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7078145Z skip: Need at least 2 CUDA devices (2.178s)
2023-01-11T21:18:21.7078151Z 
2023-01-11T21:18:21.7078339Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7078416Z Ran 1 test in 2.178s
2023-01-11T21:18:21.7078424Z 
2023-01-11T21:18:21.7078497Z OK (skipped=1)
2023-01-11T21:18:21.7078501Z 
2023-01-11T21:18:21.7078588Z Generating XML reports...
2023-01-11T21:18:21.7078932Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211239.xml
2023-01-11T21:18:21.7079166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7079445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7079571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7079842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7079978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7079983Z 
2023-01-11T21:18:21.7080058Z Running tests...
2023-01-11T21:18:21.7080260Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7080501Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21511
2023-01-11T21:18:21.7080661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21512
2023-01-11T21:18:21.7080818Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21513
2023-01-11T21:18:21.7081094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7081219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7081489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7081701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7081884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7082239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7082390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7082675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7082813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7082994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7083267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7083378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7083662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7083801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7083983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7084286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7084584Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7084881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7085048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7085211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7085360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7085470Z skip: Need at least 2 CUDA devices (2.188s)
2023-01-11T21:18:21.7085476Z 
2023-01-11T21:18:21.7085678Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7085755Z Ran 1 test in 2.188s
2023-01-11T21:18:21.7085760Z 
2023-01-11T21:18:21.7085834Z OK (skipped=1)
2023-01-11T21:18:21.7085839Z 
2023-01-11T21:18:21.7085925Z Generating XML reports...
2023-01-11T21:18:21.7086273Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211243.xml
2023-01-11T21:18:21.7086508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7086769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7086895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7087180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7087317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7087325Z 
2023-01-11T21:18:21.7087400Z Running tests...
2023-01-11T21:18:21.7087597Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7087844Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21564
2023-01-11T21:18:21.7088004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21565
2023-01-11T21:18:21.7088159Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21566
2023-01-11T21:18:21.7088420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7088545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7088862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7089025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7089205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7089484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7089608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7089890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7090013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7090191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7090464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7090588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7090871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7091006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7091183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7091485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7091784Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7092065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7092235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7092398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7092568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7092675Z skip: Need at least 2 CUDA devices (2.172s)
2023-01-11T21:18:21.7092681Z 
2023-01-11T21:18:21.7092877Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7092955Z Ran 1 test in 2.172s
2023-01-11T21:18:21.7092961Z 
2023-01-11T21:18:21.7093034Z OK (skipped=1)
2023-01-11T21:18:21.7093039Z 
2023-01-11T21:18:21.7093124Z Generating XML reports...
2023-01-11T21:18:21.7093452Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211247.xml
2023-01-11T21:18:21.7093687Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7093963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7094090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7094371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7094508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7094514Z 
2023-01-11T21:18:21.7094587Z Running tests...
2023-01-11T21:18:21.7094783Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7095485Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.256s)
2023-01-11T21:18:21.7095520Z 
2023-01-11T21:18:21.7095745Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7095809Z Ran 1 test in 0.257s
2023-01-11T21:18:21.7095815Z 
2023-01-11T21:18:21.7095888Z OK (skipped=1)
2023-01-11T21:18:21.7095893Z 
2023-01-11T21:18:21.7095982Z Generating XML reports...
2023-01-11T21:18:21.7096325Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211251.xml
2023-01-11T21:18:21.7096560Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7096836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7096961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7097247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7097372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7097391Z 
2023-01-11T21:18:21.7097453Z Running tests...
2023-01-11T21:18:21.7097649Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7098328Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.267s)
2023-01-11T21:18:21.7098334Z 
2023-01-11T21:18:21.7098530Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7098607Z Ran 1 test in 0.268s
2023-01-11T21:18:21.7098612Z 
2023-01-11T21:18:21.7098684Z OK (skipped=1)
2023-01-11T21:18:21.7098692Z 
2023-01-11T21:18:21.7098778Z Generating XML reports...
2023-01-11T21:18:21.7099122Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211253.xml
2023-01-11T21:18:21.7099356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7099622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7099748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7100031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7100170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7100175Z 
2023-01-11T21:18:21.7100250Z Running tests...
2023-01-11T21:18:21.7100446Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7101164Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.264s)
2023-01-11T21:18:21.7101172Z 
2023-01-11T21:18:21.7101367Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7101444Z Ran 1 test in 0.264s
2023-01-11T21:18:21.7101450Z 
2023-01-11T21:18:21.7101522Z OK (skipped=1)
2023-01-11T21:18:21.7101528Z 
2023-01-11T21:18:21.7101600Z Generating XML reports...
2023-01-11T21:18:21.7102059Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211255.xml
2023-01-11T21:18:21.7102291Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7102622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7102786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7103075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7103212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7103217Z 
2023-01-11T21:18:21.7103291Z Running tests...
2023-01-11T21:18:21.7103488Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7103838Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s)
2023-01-11T21:18:21.7103858Z 
2023-01-11T21:18:21.7104040Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7104120Z Ran 1 test in 0.003s
2023-01-11T21:18:21.7104126Z 
2023-01-11T21:18:21.7104199Z OK (skipped=1)
2023-01-11T21:18:21.7104203Z 
2023-01-11T21:18:21.7104289Z Generating XML reports...
2023-01-11T21:18:21.7104631Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211257.xml
2023-01-11T21:18:21.7104867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7105139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7105264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7105534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7105671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7105677Z 
2023-01-11T21:18:21.7105751Z Running tests...
2023-01-11T21:18:21.7105951Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7106193Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21657
2023-01-11T21:18:21.7106351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21658
2023-01-11T21:18:21.7106507Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21659
2023-01-11T21:18:21.7106781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7106908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7107180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7107318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7107497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7107804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7108078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7108201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7108485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7108620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7108783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7109056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7109225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7109536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7109673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7109850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7110153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7110453Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7110619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7110769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7110931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7111037Z skip: Need at least 3 CUDA devices (2.179s)
2023-01-11T21:18:21.7111043Z 
2023-01-11T21:18:21.7111243Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7111323Z Ran 1 test in 2.179s
2023-01-11T21:18:21.7111329Z 
2023-01-11T21:18:21.7111402Z OK (skipped=1)
2023-01-11T21:18:21.7111407Z 
2023-01-11T21:18:21.7111492Z Generating XML reports...
2023-01-11T21:18:21.7111832Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211259.xml
2023-01-11T21:18:21.7112068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7112330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7112456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7112742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7112883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7112888Z 
2023-01-11T21:18:21.7112962Z Running tests...
2023-01-11T21:18:21.7113158Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7113390Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21710
2023-01-11T21:18:21.7113550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21711
2023-01-11T21:18:21.7113696Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21712
2023-01-11T21:18:21.7113969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7114091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7114375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7114512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7114690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7114964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7115088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7115368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7115491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7115668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7115999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7116300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7116426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7116711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7116846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7117023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7117318Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7117605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7117775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7117943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7118107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7118282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7118455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7118634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7118931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7119226Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7119509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7119577Z ok (2.176s)
2023-01-11T21:18:21.7119583Z 
2023-01-11T21:18:21.7119782Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7119860Z Ran 1 test in 2.176s
2023-01-11T21:18:21.7119866Z 
2023-01-11T21:18:21.7119928Z OK
2023-01-11T21:18:21.7119933Z 
2023-01-11T21:18:21.7120017Z Generating XML reports...
2023-01-11T21:18:21.7120361Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211303.xml
2023-01-11T21:18:21.7120593Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7120855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7120986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7121275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7121408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7121414Z 
2023-01-11T21:18:21.7121489Z Running tests...
2023-01-11T21:18:21.7121748Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7121979Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21772
2023-01-11T21:18:21.7122137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21773
2023-01-11T21:18:21.7122291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21774
2023-01-11T21:18:21.7122553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7122714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7123025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7123164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7123343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7123619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7123743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7124024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7124159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7124323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7124602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7124725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7125006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7125139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7125314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7125615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7125915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7126200Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7126369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7126535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7126697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7126873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7127046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7127219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7127514Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7127813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7128104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7128160Z ok (2.375s)
2023-01-11T21:18:21.7128166Z 
2023-01-11T21:18:21.7128366Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7128441Z Ran 1 test in 2.375s
2023-01-11T21:18:21.7128447Z 
2023-01-11T21:18:21.7128509Z OK
2023-01-11T21:18:21.7128514Z 
2023-01-11T21:18:21.7128601Z Generating XML reports...
2023-01-11T21:18:21.7128944Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211307.xml
2023-01-11T21:18:21.7129180Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7129457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7129597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7129925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7130065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7130070Z 
2023-01-11T21:18:21.7130147Z Running tests...
2023-01-11T21:18:21.7130346Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7131045Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.256s)
2023-01-11T21:18:21.7131053Z 
2023-01-11T21:18:21.7131250Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7131328Z Ran 1 test in 0.256s
2023-01-11T21:18:21.7131333Z 
2023-01-11T21:18:21.7131407Z OK (skipped=1)
2023-01-11T21:18:21.7131412Z 
2023-01-11T21:18:21.7131498Z Generating XML reports...
2023-01-11T21:18:21.7131827Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211311.xml
2023-01-11T21:18:21.7132062Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7132339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7132467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7132749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7132888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7132894Z 
2023-01-11T21:18:21.7132968Z Running tests...
2023-01-11T21:18:21.7133169Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7133503Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s)
2023-01-11T21:18:21.7133509Z 
2023-01-11T21:18:21.7133688Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7133767Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7133772Z 
2023-01-11T21:18:21.7133844Z OK (skipped=1)
2023-01-11T21:18:21.7133848Z 
2023-01-11T21:18:21.7133934Z Generating XML reports...
2023-01-11T21:18:21.7134275Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211313.xml
2023-01-11T21:18:21.7134507Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7134785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7134913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7135182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7135320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7135325Z 
2023-01-11T21:18:21.7135400Z Running tests...
2023-01-11T21:18:21.7135598Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7135841Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21851
2023-01-11T21:18:21.7136002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21852
2023-01-11T21:18:21.7136158Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21853
2023-01-11T21:18:21.7136466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7136620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7136889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7137027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7137205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7137504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7137778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7137903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7138187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7138325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7138500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7138760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7138885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7139167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7139303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7139479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7139782Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7140085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7140253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7140418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7140568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7140637Z ok (2.173s)
2023-01-11T21:18:21.7140643Z 
2023-01-11T21:18:21.7140844Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7140920Z Ran 1 test in 2.174s
2023-01-11T21:18:21.7140926Z 
2023-01-11T21:18:21.7140989Z OK
2023-01-11T21:18:21.7140994Z 
2023-01-11T21:18:21.7141081Z Generating XML reports...
2023-01-11T21:18:21.7141423Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211315.xml
2023-01-11T21:18:21.7141773Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7142041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7142165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7142449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7142587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7142593Z 
2023-01-11T21:18:21.7142668Z Running tests...
2023-01-11T21:18:21.7142870Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7143095Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21904
2023-01-11T21:18:21.7143310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21905
2023-01-11T21:18:21.7143502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21906
2023-01-11T21:18:21.7143767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7143893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7144180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7144319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7144497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7144770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7144898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7145182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7145305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7145483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7145757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7145882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7146163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7146299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7146479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7146780Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7147081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7147357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7147525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7147774Z STAGE:2023-01-11 21:13:20 21904:21904 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7147936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7148100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7148511Z STAGE:2023-01-11 21:13:20 21906:21906 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:13:20 21905:21905 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7148518Z 
2023-01-11T21:18:21.7148769Z STAGE:2023-01-11 21:13:20 21906:21906 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7149015Z STAGE:2023-01-11 21:13:20 21905:21905 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7149276Z STAGE:2023-01-11 21:13:20 21906:21906 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7149537Z STAGE:2023-01-11 21:13:20 21905:21905 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7149767Z STAGE:2023-01-11 21:13:20 21904:21904 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7150028Z STAGE:2023-01-11 21:13:20 21904:21904 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7150303Z STAGE:2023-01-11 21:13:20 21906:21906 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7150570Z STAGE:2023-01-11 21:13:20 21905:21905 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7150822Z STAGE:2023-01-11 21:13:20 21906:21906 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7151078Z STAGE:2023-01-11 21:13:20 21906:21906 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7151326Z STAGE:2023-01-11 21:13:20 21906:21906 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7151568Z STAGE:2023-01-11 21:13:20 21904:21904 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7151818Z STAGE:2023-01-11 21:13:20 21904:21904 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7152059Z STAGE:2023-01-11 21:13:20 21904:21904 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7152310Z STAGE:2023-01-11 21:13:20 21905:21905 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7152565Z STAGE:2023-01-11 21:13:20 21905:21905 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7152806Z STAGE:2023-01-11 21:13:20 21904:21904 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7153051Z STAGE:2023-01-11 21:13:20 21904:21904 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7153305Z STAGE:2023-01-11 21:13:20 21904:21904 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7153547Z STAGE:2023-01-11 21:13:20 21905:21905 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7153794Z STAGE:2023-01-11 21:13:20 21905:21905 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7154051Z STAGE:2023-01-11 21:13:20 21905:21905 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7154284Z STAGE:2023-01-11 21:13:20 21906:21906 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7154541Z STAGE:2023-01-11 21:13:20 21906:21906 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7154612Z ok (2.491s)
2023-01-11T21:18:21.7154618Z 
2023-01-11T21:18:21.7154819Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7154896Z Ran 1 test in 2.491s
2023-01-11T21:18:21.7154902Z 
2023-01-11T21:18:21.7154965Z OK
2023-01-11T21:18:21.7154970Z 
2023-01-11T21:18:21.7155056Z Generating XML reports...
2023-01-11T21:18:21.7155402Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211319.xml
2023-01-11T21:18:21.7155624Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7155902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7156029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7156317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7156456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7156461Z 
2023-01-11T21:18:21.7156534Z Running tests...
2023-01-11T21:18:21.7156732Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7156967Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21960
2023-01-11T21:18:21.7157127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21961
2023-01-11T21:18:21.7157271Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21962
2023-01-11T21:18:21.7157598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7157750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7158037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7158174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7158354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7158627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7158752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7159020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7159160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7159338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7159609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7159735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7160015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7160151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7160329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7160627Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7160914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7161210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7161377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7161543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7161772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7161844Z ok (2.275s)
2023-01-11T21:18:21.7161849Z 
2023-01-11T21:18:21.7162053Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7162129Z Ran 1 test in 2.276s
2023-01-11T21:18:21.7162135Z 
2023-01-11T21:18:21.7162199Z OK
2023-01-11T21:18:21.7162204Z 
2023-01-11T21:18:21.7162278Z Generating XML reports...
2023-01-11T21:18:21.7162627Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211323.xml
2023-01-11T21:18:21.7162864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7163143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7163266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7163548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7163687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7163692Z 
2023-01-11T21:18:21.7163766Z Running tests...
2023-01-11T21:18:21.7163951Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7164141Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2023-01-11T21:18:21.7164178Z 
2023-01-11T21:18:21.7164377Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7164482Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7164489Z 
2023-01-11T21:18:21.7164562Z OK (skipped=1)
2023-01-11T21:18:21.7164567Z 
2023-01-11T21:18:21.7164652Z Generating XML reports...
2023-01-11T21:18:21.7164995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211327.xml
2023-01-11T21:18:21.7165229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7165501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7165611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7165894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7166036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7166041Z 
2023-01-11T21:18:21.7166115Z Running tests...
2023-01-11T21:18:21.7166316Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7166551Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22023
2023-01-11T21:18:21.7166710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22024
2023-01-11T21:18:21.7166866Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22025
2023-01-11T21:18:21.7167141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7167253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7167536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7167673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7167855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7168126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7168249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7168532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7168667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7168833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7169108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7169234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7169518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7169653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7169830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7170132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7170432Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7170730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7170881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7171073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7171263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7171442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7171616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7171789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7172083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7172378Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7172669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7172910Z STAGE:2023-01-11 21:13:31 22024:22024 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7173161Z STAGE:2023-01-11 21:13:31 22023:22023 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7173403Z STAGE:2023-01-11 21:13:31 22025:22025 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7173816Z STAGE:2023-01-11 21:13:31 22025:22025 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:13:31 22024:22024 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7173822Z 
2023-01-11T21:18:21.7174257Z STAGE:2023-01-11 21:13:31 22025:22025 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:13:31 22024:22024 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7174266Z 
2023-01-11T21:18:21.7174519Z STAGE:2023-01-11 21:13:31 22023:22023 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7174781Z STAGE:2023-01-11 21:13:31 22023:22023 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7175024Z STAGE:2023-01-11 21:13:31 22025:22025 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7175263Z STAGE:2023-01-11 21:13:31 22024:22024 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7175510Z STAGE:2023-01-11 21:13:31 22025:22025 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7175769Z STAGE:2023-01-11 21:13:31 22025:22025 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7175995Z STAGE:2023-01-11 21:13:31 22025:22025 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7176234Z STAGE:2023-01-11 21:13:31 22023:22023 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7176486Z STAGE:2023-01-11 21:13:31 22023:22023 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7176744Z STAGE:2023-01-11 21:13:31 22023:22023 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7176992Z STAGE:2023-01-11 21:13:31 22024:22024 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7177247Z STAGE:2023-01-11 21:13:31 22024:22024 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7177487Z STAGE:2023-01-11 21:13:31 22023:22023 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7177732Z STAGE:2023-01-11 21:13:31 22023:22023 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7177989Z STAGE:2023-01-11 21:13:31 22023:22023 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7178215Z STAGE:2023-01-11 21:13:31 22024:22024 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7178494Z STAGE:2023-01-11 21:13:31 22024:22024 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7178781Z STAGE:2023-01-11 21:13:31 22024:22024 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7179031Z STAGE:2023-01-11 21:13:31 22025:22025 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7179285Z STAGE:2023-01-11 21:13:31 22025:22025 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7179354Z ok (2.479s)
2023-01-11T21:18:21.7179360Z 
2023-01-11T21:18:21.7179559Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7179636Z Ran 1 test in 2.479s
2023-01-11T21:18:21.7179642Z 
2023-01-11T21:18:21.7179690Z OK
2023-01-11T21:18:21.7179708Z 
2023-01-11T21:18:21.7179780Z Generating XML reports...
2023-01-11T21:18:21.7180128Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211329.xml
2023-01-11T21:18:21.7180366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7180644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7180771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7181052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7181188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7181194Z 
2023-01-11T21:18:21.7181270Z Running tests...
2023-01-11T21:18:21.7181451Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7181800Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22088
2023-01-11T21:18:21.7181962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22089
2023-01-11T21:18:21.7182120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22090
2023-01-11T21:18:21.7182400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7182527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7182810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7182946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7183125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7183386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7183513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7183797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7183932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7184110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7184381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7184507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7184790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7184912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7185087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7185439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7185804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7186105Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7186271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7186437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7186598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7186773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7186935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7187110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7187411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7187709Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7188122Z STAGE:2023-01-11 21:13:35 22090:22090 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:13:35 22089:22089 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7188128Z 
2023-01-11T21:18:21.7188381Z STAGE:2023-01-11 21:13:35 22090:22090 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7188640Z STAGE:2023-01-11 21:13:35 22090:22090 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7188892Z STAGE:2023-01-11 21:13:35 22089:22089 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7189150Z STAGE:2023-01-11 21:13:35 22089:22089 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7189395Z STAGE:2023-01-11 21:13:35 22090:22090 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7189621Z STAGE:2023-01-11 21:13:35 22089:22089 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7189871Z STAGE:2023-01-11 21:13:35 22089:22089 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7190125Z STAGE:2023-01-11 21:13:35 22089:22089 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7190375Z STAGE:2023-01-11 21:13:35 22090:22090 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7190629Z STAGE:2023-01-11 21:13:35 22090:22090 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7190931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7191002Z ok (2.391s)
2023-01-11T21:18:21.7191008Z 
2023-01-11T21:18:21.7191207Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7191284Z Ran 1 test in 2.391s
2023-01-11T21:18:21.7191290Z 
2023-01-11T21:18:21.7191338Z OK
2023-01-11T21:18:21.7191342Z 
2023-01-11T21:18:21.7191428Z Generating XML reports...
2023-01-11T21:18:21.7191778Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211333.xml
2023-01-11T21:18:21.7192014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7192293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7192449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7192761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7192900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7192905Z 
2023-01-11T21:18:21.7192979Z Running tests...
2023-01-11T21:18:21.7193166Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7193397Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22149
2023-01-11T21:18:21.7193560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22150
2023-01-11T21:18:21.7193717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22151
2023-01-11T21:18:21.7193990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7194116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7194400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7194538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7194704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7194978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7195102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7195388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7195524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7195704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7196009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7196282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7196407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7196672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7196810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7196988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7197287Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7197586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7197755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7197921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7198086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7198157Z ok (2.177s)
2023-01-11T21:18:21.7198162Z 
2023-01-11T21:18:21.7198348Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7198425Z Ran 1 test in 2.177s
2023-01-11T21:18:21.7198431Z 
2023-01-11T21:18:21.7198493Z OK
2023-01-11T21:18:21.7198498Z 
2023-01-11T21:18:21.7198584Z Generating XML reports...
2023-01-11T21:18:21.7198928Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211337.xml
2023-01-11T21:18:21.7199195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7199504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7199630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7199896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7200036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7200042Z 
2023-01-11T21:18:21.7200117Z Running tests...
2023-01-11T21:18:21.7200317Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7201006Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.256s)
2023-01-11T21:18:21.7201015Z 
2023-01-11T21:18:21.7201217Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7201293Z Ran 1 test in 0.256s
2023-01-11T21:18:21.7201299Z 
2023-01-11T21:18:21.7201373Z OK (skipped=1)
2023-01-11T21:18:21.7201378Z 
2023-01-11T21:18:21.7201464Z Generating XML reports...
2023-01-11T21:18:21.7201876Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211341.xml
2023-01-11T21:18:21.7202101Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7202379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7202506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7202793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7202933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7202938Z 
2023-01-11T21:18:21.7203013Z Running tests...
2023-01-11T21:18:21.7203210Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7203442Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22212
2023-01-11T21:18:21.7203603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22213
2023-01-11T21:18:21.7203746Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22214
2023-01-11T21:18:21.7204020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7204151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7204437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7204574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7204754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7205027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7205151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7205420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7205557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7205736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7206055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7206180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7206485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7206624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7206800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7207103Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7207387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7207681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7207851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7208030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7208196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7208359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7208531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7208706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7209005Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7209287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7209588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7209659Z ok (2.377s)
2023-01-11T21:18:21.7209665Z 
2023-01-11T21:18:21.7209863Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7209941Z Ran 1 test in 2.377s
2023-01-11T21:18:21.7209946Z 
2023-01-11T21:18:21.7210008Z OK
2023-01-11T21:18:21.7210014Z 
2023-01-11T21:18:21.7210099Z Generating XML reports...
2023-01-11T21:18:21.7210442Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211343.xml
2023-01-11T21:18:21.7210678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7210942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7211071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7211356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7211498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7211505Z 
2023-01-11T21:18:21.7211580Z Running tests...
2023-01-11T21:18:21.7211779Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7212006Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22271
2023-01-11T21:18:21.7212167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22272
2023-01-11T21:18:21.7212308Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22273
2023-01-11T21:18:21.7212584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7212741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7213053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7213192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7213370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7213643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7213767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7214047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7214169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7214350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7214625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7214750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7215037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7215173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7215351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7215649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7215947Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7216232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7216403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7216567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7216731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7216801Z ok (2.286s)
2023-01-11T21:18:21.7216807Z 
2023-01-11T21:18:21.7217006Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7217085Z Ran 1 test in 2.286s
2023-01-11T21:18:21.7217090Z 
2023-01-11T21:18:21.7217153Z OK
2023-01-11T21:18:21.7217158Z 
2023-01-11T21:18:21.7217229Z Generating XML reports...
2023-01-11T21:18:21.7217574Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211347.xml
2023-01-11T21:18:21.7217811Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7218095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7218221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7218504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7218643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7218648Z 
2023-01-11T21:18:21.7218724Z Running tests...
2023-01-11T21:18:21.7218924Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7219135Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22324
2023-01-11T21:18:21.7219293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22325
2023-01-11T21:18:21.7219478Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22326
2023-01-11T21:18:21.7219785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7219911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7220194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7220329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7220508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7220767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7220891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7221170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7221310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7221490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7221884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7222009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7222294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7222429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7222592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7222896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7223200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7223496Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7223663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7223830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7223994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7224063Z ok (2.283s)
2023-01-11T21:18:21.7224068Z 
2023-01-11T21:18:21.7224268Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7224332Z Ran 1 test in 2.283s
2023-01-11T21:18:21.7224337Z 
2023-01-11T21:18:21.7224398Z OK
2023-01-11T21:18:21.7224406Z 
2023-01-11T21:18:21.7224490Z Generating XML reports...
2023-01-11T21:18:21.7224835Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211352.xml
2023-01-11T21:18:21.7225070Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7225345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7225471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7225755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7225877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7225898Z 
2023-01-11T21:18:21.7225960Z Running tests...
2023-01-11T21:18:21.7226158Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7226446Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22377
2023-01-11T21:18:21.7226641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22378
2023-01-11T21:18:21.7226801Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22379
2023-01-11T21:18:21.7227190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7227316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7227599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7227722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7227902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7228179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7228305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7228589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7228725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7228904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7229178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7229304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7229572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7229711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7229889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7230191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7230492Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7230786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7230956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7231117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7231283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7231449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7231626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7231800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7232095Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7232391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7232698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7232794Z ok (2.171s)
2023-01-11T21:18:21.7232802Z 
2023-01-11T21:18:21.7233040Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7233144Z Ran 1 test in 2.172s
2023-01-11T21:18:21.7233164Z 
2023-01-11T21:18:21.7233215Z OK
2023-01-11T21:18:21.7233219Z 
2023-01-11T21:18:21.7233304Z Generating XML reports...
2023-01-11T21:18:21.7233679Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211356.xml
2023-01-11T21:18:21.7233916Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7234193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7234319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7234603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7234738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7234743Z 
2023-01-11T21:18:21.7234805Z Running tests...
2023-01-11T21:18:21.7235009Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7235249Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22439
2023-01-11T21:18:21.7235408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22440
2023-01-11T21:18:21.7235565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22441
2023-01-11T21:18:21.7235840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7235965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7236246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7236383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7236548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7236828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7236951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7237234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7237370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7237549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7237846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7238119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7238231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7238513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7238651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7238828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7239123Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7239421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7239585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7239749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7239912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7240103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7240307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7240483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7240781Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7241076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7241364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7241433Z ok (2.281s)
2023-01-11T21:18:21.7241439Z 
2023-01-11T21:18:21.7241713Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7241790Z Ran 1 test in 2.282s
2023-01-11T21:18:21.7241795Z 
2023-01-11T21:18:21.7241843Z OK
2023-01-11T21:18:21.7241848Z 
2023-01-11T21:18:21.7241935Z Generating XML reports...
2023-01-11T21:18:21.7242283Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211400.xml
2023-01-11T21:18:21.7242516Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7242794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7242919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7243204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7243340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7243348Z 
2023-01-11T21:18:21.7243423Z Running tests...
2023-01-11T21:18:21.7243609Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7243939Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s)
2023-01-11T21:18:21.7243946Z 
2023-01-11T21:18:21.7244143Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7244220Z Ran 1 test in 0.003s
2023-01-11T21:18:21.7244225Z 
2023-01-11T21:18:21.7244300Z OK (skipped=1)
2023-01-11T21:18:21.7244305Z 
2023-01-11T21:18:21.7244391Z Generating XML reports...
2023-01-11T21:18:21.7244733Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211404.xml
2023-01-11T21:18:21.7244965Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7245243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7245357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7245643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7245779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7245784Z 
2023-01-11T21:18:21.7245858Z Running tests...
2023-01-11T21:18:21.7246054Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7246276Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22508
2023-01-11T21:18:21.7246435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22509
2023-01-11T21:18:21.7246591Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22510
2023-01-11T21:18:21.7246889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7247017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7247328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7247466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7247643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7247918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7248041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7248329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7248464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7248629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7248906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7249029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7249310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7249446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7249619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7249924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7250223Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7250521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7250674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7250840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7251003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7251073Z ok (2.278s)
2023-01-11T21:18:21.7251079Z 
2023-01-11T21:18:21.7251277Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7251353Z Ran 1 test in 2.278s
2023-01-11T21:18:21.7251360Z 
2023-01-11T21:18:21.7251422Z OK
2023-01-11T21:18:21.7251427Z 
2023-01-11T21:18:21.7251514Z Generating XML reports...
2023-01-11T21:18:21.7251841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211405.xml
2023-01-11T21:18:21.7252075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7252355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7252481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7252765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7252902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7252908Z 
2023-01-11T21:18:21.7252984Z Running tests...
2023-01-11T21:18:21.7253183Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7253407Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22561
2023-01-11T21:18:21.7253603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22562
2023-01-11T21:18:21.7253790Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22563
2023-01-11T21:18:21.7254068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7254194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7254477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7254614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7254794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7255068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7255181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7255463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7255600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7255779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7256051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7256175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7256455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7256591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7256769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7257059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7257363Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7257656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7257822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7257983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7258148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7258218Z ok (2.284s)
2023-01-11T21:18:21.7258224Z 
2023-01-11T21:18:21.7258422Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7258502Z Ran 1 test in 2.285s
2023-01-11T21:18:21.7258508Z 
2023-01-11T21:18:21.7258557Z OK
2023-01-11T21:18:21.7258562Z 
2023-01-11T21:18:21.7258648Z Generating XML reports...
2023-01-11T21:18:21.7258994Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211410.xml
2023-01-11T21:18:21.7259231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7259507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7259633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7259919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7260056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7260062Z 
2023-01-11T21:18:21.7260123Z Running tests...
2023-01-11T21:18:21.7260358Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7260634Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22614
2023-01-11T21:18:21.7260793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22615
2023-01-11T21:18:21.7260952Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22616
2023-01-11T21:18:21.7261231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7261356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7261640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7261871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7262038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7262316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7262444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7262724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7262861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7263039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7263314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7263439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7263707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7263846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7264022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7264324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7264622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7264920Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7265085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7265250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7265414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7265655Z STAGE:2023-01-11 21:14:15 22615:22615 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7265900Z STAGE:2023-01-11 21:14:15 22614:22614 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7266140Z STAGE:2023-01-11 21:14:15 22616:22616 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7266389Z STAGE:2023-01-11 21:14:15 22614:22614 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7266648Z STAGE:2023-01-11 21:14:15 22614:22614 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7266900Z STAGE:2023-01-11 21:14:15 22615:22615 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7267159Z STAGE:2023-01-11 21:14:15 22615:22615 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7267461Z STAGE:2023-01-11 21:14:15 22616:22616 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7267752Z STAGE:2023-01-11 21:14:15 22616:22616 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7267810Z ok (2.272s)
2023-01-11T21:18:21.7267830Z 
2023-01-11T21:18:21.7268019Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7268097Z Ran 1 test in 2.273s
2023-01-11T21:18:21.7268103Z 
2023-01-11T21:18:21.7268165Z OK
2023-01-11T21:18:21.7268170Z 
2023-01-11T21:18:21.7268255Z Generating XML reports...
2023-01-11T21:18:21.7268603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211414.xml
2023-01-11T21:18:21.7268839Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7269116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7269244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7269516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7269655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7269661Z 
2023-01-11T21:18:21.7269734Z Running tests...
2023-01-11T21:18:21.7269934Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7270172Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22670
2023-01-11T21:18:21.7270332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22671
2023-01-11T21:18:21.7270489Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22672
2023-01-11T21:18:21.7270764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7270877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7271162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7271302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7271481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7271758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7271882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7272164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7272303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7272484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7272744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7272868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7273148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7273283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7273460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7273762Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7274060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7274384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7274577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7274730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7274896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7275147Z STAGE:2023-01-11 21:14:20 22672:22672 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7275388Z STAGE:2023-01-11 21:14:20 22670:22670 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7275632Z STAGE:2023-01-11 21:14:20 22671:22671 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7275881Z STAGE:2023-01-11 21:14:20 22670:22670 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7276142Z STAGE:2023-01-11 21:14:20 22670:22670 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7276392Z STAGE:2023-01-11 21:14:20 22671:22671 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7276651Z STAGE:2023-01-11 21:14:20 22671:22671 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7276888Z STAGE:2023-01-11 21:14:20 22672:22672 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7277145Z STAGE:2023-01-11 21:14:20 22672:22672 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7277214Z ok (2.277s)
2023-01-11T21:18:21.7277220Z 
2023-01-11T21:18:21.7277419Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7277495Z Ran 1 test in 2.278s
2023-01-11T21:18:21.7277500Z 
2023-01-11T21:18:21.7277563Z OK
2023-01-11T21:18:21.7277568Z 
2023-01-11T21:18:21.7277655Z Generating XML reports...
2023-01-11T21:18:21.7277997Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211418.xml
2023-01-11T21:18:21.7278220Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7278497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7278621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7278906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7279043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7279048Z 
2023-01-11T21:18:21.7279122Z Running tests...
2023-01-11T21:18:21.7279320Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7279663Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s)
2023-01-11T21:18:21.7279671Z 
2023-01-11T21:18:21.7279871Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7279934Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7279954Z 
2023-01-11T21:18:21.7280016Z OK (skipped=1)
2023-01-11T21:18:21.7280021Z 
2023-01-11T21:18:21.7280106Z Generating XML reports...
2023-01-11T21:18:21.7280448Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211422.xml
2023-01-11T21:18:21.7280683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7280957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7281082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7281398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7281533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7281636Z 
2023-01-11T21:18:21.7281713Z Running tests...
2023-01-11T21:18:21.7281914Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7282320Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s)
2023-01-11T21:18:21.7282326Z 
2023-01-11T21:18:21.7282524Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7282601Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7282607Z 
2023-01-11T21:18:21.7282679Z OK (skipped=1)
2023-01-11T21:18:21.7282684Z 
2023-01-11T21:18:21.7282770Z Generating XML reports...
2023-01-11T21:18:21.7283110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211424.xml
2023-01-11T21:18:21.7283345Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7283609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7283735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7284017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7284154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7284160Z 
2023-01-11T21:18:21.7284237Z Running tests...
2023-01-11T21:18:21.7284438Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7284688Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22746
2023-01-11T21:18:21.7284849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22747
2023-01-11T21:18:21.7285006Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22748
2023-01-11T21:18:21.7285269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7285395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7285677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7285813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7285991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7286265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7286393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7286676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7286800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7286979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7287250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7287376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7287656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7287792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7287968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7288308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7288634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7288917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7289085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7289248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7289410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7289669Z [E ProcessGroupGloo.cpp:138] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2023-01-11T21:18:21.7289840Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 2000 ms
2023-01-11T21:18:21.7289909Z ok (4.274s)
2023-01-11T21:18:21.7289915Z 
2023-01-11T21:18:21.7290118Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7290194Z Ran 1 test in 4.275s
2023-01-11T21:18:21.7290200Z 
2023-01-11T21:18:21.7290248Z OK
2023-01-11T21:18:21.7290253Z 
2023-01-11T21:18:21.7290341Z Generating XML reports...
2023-01-11T21:18:21.7290684Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211425.xml
2023-01-11T21:18:21.7290918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7291194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7291320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7291605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7291743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7291749Z 
2023-01-11T21:18:21.7291822Z Running tests...
2023-01-11T21:18:21.7292009Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7292247Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22799
2023-01-11T21:18:21.7292405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22800
2023-01-11T21:18:21.7292561Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22801
2023-01-11T21:18:21.7292837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7292963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7293251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7293389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7293552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7293827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7293952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7294236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7294372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7294549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7294888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7295038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7295320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7295442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7295617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7295916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7296217Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7296514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7296680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7296847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7297011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7297178Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2023-01-11T21:18:21.7297425Z [E ProcessGroupGloo.cpp:138] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2023-01-11T21:18:21.7297497Z ok (4.382s)
2023-01-11T21:18:21.7297504Z 
2023-01-11T21:18:21.7297705Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7297783Z Ran 1 test in 4.382s
2023-01-11T21:18:21.7297789Z 
2023-01-11T21:18:21.7297854Z OK
2023-01-11T21:18:21.7297859Z 
2023-01-11T21:18:21.7297945Z Generating XML reports...
2023-01-11T21:18:21.7298291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211431.xml
2023-01-11T21:18:21.7298526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7298789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7298917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7299258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7299394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7299400Z 
2023-01-11T21:18:21.7299475Z Running tests...
2023-01-11T21:18:21.7299671Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7299925Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22852
2023-01-11T21:18:21.7300087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22853
2023-01-11T21:18:21.7300243Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22854
2023-01-11T21:18:21.7300506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7300630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7300911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7301046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7301225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7301575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7302017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7302142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7302422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7302543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7302719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7302995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7303119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7303403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7303540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7303717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7304020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7304322Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7304475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7304639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7304803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7304981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7305160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7305332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7305631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7305926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7306221Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7306366Z [E ProcessGroupGloo.cpp:138] Rank 0 timed out in monitoredBarrier after 0 ms.
2023-01-11T21:18:21.7306491Z No ranks successfully processed in monitoredBarrier.
2023-01-11T21:18:21.7306656Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2023-01-11T21:18:21.7306725Z ok (2.273s)
2023-01-11T21:18:21.7306732Z 
2023-01-11T21:18:21.7306935Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7307014Z Ran 1 test in 2.273s
2023-01-11T21:18:21.7307020Z 
2023-01-11T21:18:21.7307076Z OK
2023-01-11T21:18:21.7307082Z 
2023-01-11T21:18:21.7307170Z Generating XML reports...
2023-01-11T21:18:21.7307511Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211438.xml
2023-01-11T21:18:21.7307748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7308024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7308149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7308461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7308629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7308635Z 
2023-01-11T21:18:21.7308707Z Running tests...
2023-01-11T21:18:21.7308906Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7309154Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22914
2023-01-11T21:18:21.7309314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22915
2023-01-11T21:18:21.7309469Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22916
2023-01-11T21:18:21.7309746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7309870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7310139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7310279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7310457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7310733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7310857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7311141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7311276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7311454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7311717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7311842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7312124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7312259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7312436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7312739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7313041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7313339Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7313506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7313661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7313825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7314001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7314173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7314343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7314638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7314934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7315282Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7315453Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:18:21.7315830Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:475: UserWarning: Running monitored_barrier on global rank 2 which does not belong to the given group.
2023-01-11T21:18:21.7315950Z   f"Running {op_name} on global rank {global_rank} which does not "
2023-01-11T21:18:21.7316023Z ok (2.374s)
2023-01-11T21:18:21.7316029Z 
2023-01-11T21:18:21.7316230Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7316309Z Ran 1 test in 2.374s
2023-01-11T21:18:21.7316315Z 
2023-01-11T21:18:21.7316376Z OK
2023-01-11T21:18:21.7316381Z 
2023-01-11T21:18:21.7316466Z Generating XML reports...
2023-01-11T21:18:21.7316812Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211442.xml
2023-01-11T21:18:21.7317047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7317311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7317437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7317719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7317856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7317861Z 
2023-01-11T21:18:21.7317935Z Running tests...
2023-01-11T21:18:21.7318133Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7318378Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22973
2023-01-11T21:18:21.7318538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22974
2023-01-11T21:18:21.7318692Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22975
2023-01-11T21:18:21.7318954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7319079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7319360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7319495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7319669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7319942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7320066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7320346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7320467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7320641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7320911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7321035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7321316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7321449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7321729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7322066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7322366Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7322646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7322814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7322980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7323141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7323306Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:18:21.7323472Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:18:21.7323636Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1, 2 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:18:21.7323705Z ok (2.372s)
2023-01-11T21:18:21.7323710Z 
2023-01-11T21:18:21.7323910Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7323974Z Ran 1 test in 2.372s
2023-01-11T21:18:21.7323979Z 
2023-01-11T21:18:21.7324040Z OK
2023-01-11T21:18:21.7324045Z 
2023-01-11T21:18:21.7324131Z Generating XML reports...
2023-01-11T21:18:21.7324477Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211446.xml
2023-01-11T21:18:21.7324710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7324986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7325114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7325396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7325518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7325535Z 
2023-01-11T21:18:21.7325595Z Running tests...
2023-01-11T21:18:21.7325790Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7326095Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T21:18:21.7326100Z 
2023-01-11T21:18:21.7326290Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7326364Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7326370Z 
2023-01-11T21:18:21.7326442Z OK (skipped=1)
2023-01-11T21:18:21.7326449Z 
2023-01-11T21:18:21.7326531Z Generating XML reports...
2023-01-11T21:18:21.7326869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211450.xml
2023-01-11T21:18:21.7327091Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7327363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7327485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7327768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7327902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7327909Z 
2023-01-11T21:18:21.7327980Z Running tests...
2023-01-11T21:18:21.7328177Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7328508Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T21:18:21.7328514Z 
2023-01-11T21:18:21.7328736Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7328811Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7328817Z 
2023-01-11T21:18:21.7328887Z OK (skipped=1)
2023-01-11T21:18:21.7328892Z 
2023-01-11T21:18:21.7328975Z Generating XML reports...
2023-01-11T21:18:21.7329310Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211452.xml
2023-01-11T21:18:21.7329542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7329817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7329940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7330222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7330357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7330364Z 
2023-01-11T21:18:21.7330424Z Running tests...
2023-01-11T21:18:21.7330619Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7330922Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T21:18:21.7330928Z 
2023-01-11T21:18:21.7331120Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7331194Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7331199Z 
2023-01-11T21:18:21.7331270Z OK (skipped=1)
2023-01-11T21:18:21.7331275Z 
2023-01-11T21:18:21.7331357Z Generating XML reports...
2023-01-11T21:18:21.7331694Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211454.xml
2023-01-11T21:18:21.7331915Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7332187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7332309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7332591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7332727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7332732Z 
2023-01-11T21:18:21.7332804Z Running tests...
2023-01-11T21:18:21.7333000Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7333300Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T21:18:21.7333306Z 
2023-01-11T21:18:21.7333500Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7333562Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7333578Z 
2023-01-11T21:18:21.7333637Z OK (skipped=1)
2023-01-11T21:18:21.7333642Z 
2023-01-11T21:18:21.7333727Z Generating XML reports...
2023-01-11T21:18:21.7334064Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211455.xml
2023-01-11T21:18:21.7334295Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7334568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7334693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7334973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7335106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7335143Z 
2023-01-11T21:18:21.7335204Z Running tests...
2023-01-11T21:18:21.7335401Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7335646Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2023-01-11T21:18:21.7335653Z 
2023-01-11T21:18:21.7335847Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7335921Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7335927Z 
2023-01-11T21:18:21.7335998Z OK (skipped=1)
2023-01-11T21:18:21.7336003Z 
2023-01-11T21:18:21.7336087Z Generating XML reports...
2023-01-11T21:18:21.7336424Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211457.xml
2023-01-11T21:18:21.7336655Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7336917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7337041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7337325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7337461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7337467Z 
2023-01-11T21:18:21.7337538Z Running tests...
2023-01-11T21:18:21.7337733Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7337910Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:18:21.7337916Z 
2023-01-11T21:18:21.7338112Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7338174Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7338191Z 
2023-01-11T21:18:21.7338249Z OK (skipped=1)
2023-01-11T21:18:21.7338256Z 
2023-01-11T21:18:21.7338339Z Generating XML reports...
2023-01-11T21:18:21.7338672Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211459.xml
2023-01-11T21:18:21.7338900Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7339172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7339294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7339573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7339706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7339712Z 
2023-01-11T21:18:21.7339770Z Running tests...
2023-01-11T21:18:21.7339968Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7340165Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:18:21.7340171Z 
2023-01-11T21:18:21.7340363Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7340437Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7340443Z 
2023-01-11T21:18:21.7340513Z OK (skipped=1)
2023-01-11T21:18:21.7340519Z 
2023-01-11T21:18:21.7340601Z Generating XML reports...
2023-01-11T21:18:21.7340939Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211500.xml
2023-01-11T21:18:21.7341167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7341428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7341551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7341950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7342141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7342182Z 
2023-01-11T21:18:21.7342256Z Running tests...
2023-01-11T21:18:21.7342452Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7342678Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:18:21.7342684Z 
2023-01-11T21:18:21.7342878Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7342955Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7342961Z 
2023-01-11T21:18:21.7343018Z OK (skipped=1)
2023-01-11T21:18:21.7343023Z 
2023-01-11T21:18:21.7343106Z Generating XML reports...
2023-01-11T21:18:21.7343441Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211502.xml
2023-01-11T21:18:21.7343677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7343955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7344079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7344357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7344492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7344497Z 
2023-01-11T21:18:21.7344572Z Running tests...
2023-01-11T21:18:21.7344752Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7345018Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23106
2023-01-11T21:18:21.7345178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23107
2023-01-11T21:18:21.7345332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23108
2023-01-11T21:18:21.7345605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7345728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7346009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7346143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7346305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7346578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7346700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7346987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7347122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7347299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7347570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7347694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7347976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7348097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7348270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7348617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7348942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7349240Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7349404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7349567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7349731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7349824Z skip: CUDA is not available. (2.174s)
2023-01-11T21:18:21.7349830Z 
2023-01-11T21:18:21.7350013Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7350090Z Ran 1 test in 2.174s
2023-01-11T21:18:21.7350096Z 
2023-01-11T21:18:21.7350166Z OK (skipped=1)
2023-01-11T21:18:21.7350171Z 
2023-01-11T21:18:21.7350255Z Generating XML reports...
2023-01-11T21:18:21.7350599Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211504.xml
2023-01-11T21:18:21.7350830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7351102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7351227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7351495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7351634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7351639Z 
2023-01-11T21:18:21.7351716Z Running tests...
2023-01-11T21:18:21.7351917Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7352179Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23159
2023-01-11T21:18:21.7352339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23160
2023-01-11T21:18:21.7352491Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23161
2023-01-11T21:18:21.7352764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7352886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7353153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7353286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7353464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7353737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7353860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7354141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7354273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7354449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7354720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7354830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7355147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7355280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7355483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7355784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7356082Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7356374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7356540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7356703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7356854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7356948Z skip: CUDA is not available. (2.273s)
2023-01-11T21:18:21.7356956Z 
2023-01-11T21:18:21.7357155Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7357229Z Ran 1 test in 2.274s
2023-01-11T21:18:21.7357235Z 
2023-01-11T21:18:21.7357307Z OK (skipped=1)
2023-01-11T21:18:21.7357312Z 
2023-01-11T21:18:21.7357395Z Generating XML reports...
2023-01-11T21:18:21.7357737Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211508.xml
2023-01-11T21:18:21.7357969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7358230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7358357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7358639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7358775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7358780Z 
2023-01-11T21:18:21.7358854Z Running tests...
2023-01-11T21:18:21.7359049Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7359249Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:18:21.7359254Z 
2023-01-11T21:18:21.7359448Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7359521Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7359527Z 
2023-01-11T21:18:21.7359585Z OK (skipped=1)
2023-01-11T21:18:21.7359589Z 
2023-01-11T21:18:21.7359672Z Generating XML reports...
2023-01-11T21:18:21.7360009Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211512.xml
2023-01-11T21:18:21.7360241Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7360516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7360641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7360920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7361054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7361059Z 
2023-01-11T21:18:21.7361131Z Running tests...
2023-01-11T21:18:21.7361313Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7361532Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.001s)
2023-01-11T21:18:21.7361656Z 
2023-01-11T21:18:21.7361856Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7361934Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7361972Z 
2023-01-11T21:18:21.7362045Z OK (skipped=1)
2023-01-11T21:18:21.7362050Z 
2023-01-11T21:18:21.7362135Z Generating XML reports...
2023-01-11T21:18:21.7362473Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211513.xml
2023-01-11T21:18:21.7362708Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7362982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7363093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7363379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7363519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7363524Z 
2023-01-11T21:18:21.7363598Z Running tests...
2023-01-11T21:18:21.7363796Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7364513Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.265s)
2023-01-11T21:18:21.7364519Z 
2023-01-11T21:18:21.7364712Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7364791Z Ran 1 test in 0.265s
2023-01-11T21:18:21.7364797Z 
2023-01-11T21:18:21.7364868Z OK (skipped=1)
2023-01-11T21:18:21.7364873Z 
2023-01-11T21:18:21.7364945Z Generating XML reports...
2023-01-11T21:18:21.7365290Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211515.xml
2023-01-11T21:18:21.7365524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7365799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7365925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7366208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7366344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7366350Z 
2023-01-11T21:18:21.7366424Z Running tests...
2023-01-11T21:18:21.7366622Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7366858Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23242
2023-01-11T21:18:21.7367023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23243
2023-01-11T21:18:21.7367181Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23244
2023-01-11T21:18:21.7367455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7367581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7367862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7367997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7368175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7368452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7368591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7368908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7369044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7369220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7369494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7369617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7369898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7370031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7370195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7370501Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7370803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7371097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7371263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7371430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7371592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7371695Z skip: Need at least 2 CUDA devices (2.282s)
2023-01-11T21:18:21.7371703Z 
2023-01-11T21:18:21.7371904Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7371968Z Ran 1 test in 2.283s
2023-01-11T21:18:21.7371987Z 
2023-01-11T21:18:21.7372048Z OK (skipped=1)
2023-01-11T21:18:21.7372054Z 
2023-01-11T21:18:21.7372140Z Generating XML reports...
2023-01-11T21:18:21.7372481Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211517.xml
2023-01-11T21:18:21.7372713Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7372990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7373116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7373399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7373536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7373542Z 
2023-01-11T21:18:21.7373602Z Running tests...
2023-01-11T21:18:21.7373802Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7374046Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23295
2023-01-11T21:18:21.7374204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23296
2023-01-11T21:18:21.7374356Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23297
2023-01-11T21:18:21.7374628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7374748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7375029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7375180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7375382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7375655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7375776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7376055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7376187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7376363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7376631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7376753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7377020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7377154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7377328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7377625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7377920Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7378210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7378373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7378533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7378698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7378789Z skip: Need at least 2 CUDA devices (2.288s)
2023-01-11T21:18:21.7378795Z 
2023-01-11T21:18:21.7378990Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7379064Z Ran 1 test in 2.288s
2023-01-11T21:18:21.7379069Z 
2023-01-11T21:18:21.7379141Z OK (skipped=1)
2023-01-11T21:18:21.7379146Z 
2023-01-11T21:18:21.7379230Z Generating XML reports...
2023-01-11T21:18:21.7379570Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211521.xml
2023-01-11T21:18:21.7379801Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7380073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7380197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7380467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7380601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7380607Z 
2023-01-11T21:18:21.7380678Z Running tests...
2023-01-11T21:18:21.7380877Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7381132Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23348
2023-01-11T21:18:21.7381288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23349
2023-01-11T21:18:21.7381441Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23350
2023-01-11T21:18:21.7381814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7381982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7382299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7382433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7382609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7382888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7383011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7383293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7383428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7383607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7383871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7383993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7384272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7384404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7384578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7384880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7385175Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7385471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7385638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7385790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7385951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7386054Z skip: Need at least 2 CUDA devices (2.182s)
2023-01-11T21:18:21.7386060Z 
2023-01-11T21:18:21.7386259Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7386335Z Ran 1 test in 2.183s
2023-01-11T21:18:21.7386341Z 
2023-01-11T21:18:21.7386413Z OK (skipped=1)
2023-01-11T21:18:21.7386418Z 
2023-01-11T21:18:21.7386506Z Generating XML reports...
2023-01-11T21:18:21.7386849Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211525.xml
2023-01-11T21:18:21.7387072Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7387344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7387466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7387745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7387878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7387883Z 
2023-01-11T21:18:21.7387955Z Running tests...
2023-01-11T21:18:21.7388151Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7388884Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2023-01-11T21:18:21.7388917Z 
2023-01-11T21:18:21.7389113Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7389187Z Ran 1 test in 0.255s
2023-01-11T21:18:21.7389192Z 
2023-01-11T21:18:21.7389250Z OK (skipped=1)
2023-01-11T21:18:21.7389266Z 
2023-01-11T21:18:21.7389338Z Generating XML reports...
2023-01-11T21:18:21.7389677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211529.xml
2023-01-11T21:18:21.7389908Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7390184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7390309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7390595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7390728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7390733Z 
2023-01-11T21:18:21.7390804Z Running tests...
2023-01-11T21:18:21.7390988Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7391710Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.258s)
2023-01-11T21:18:21.7391728Z 
2023-01-11T21:18:21.7391914Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7391988Z Ran 1 test in 0.259s
2023-01-11T21:18:21.7391993Z 
2023-01-11T21:18:21.7392064Z OK (skipped=1)
2023-01-11T21:18:21.7392070Z 
2023-01-11T21:18:21.7392154Z Generating XML reports...
2023-01-11T21:18:21.7392492Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211531.xml
2023-01-11T21:18:21.7392724Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7392996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7393119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7393388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7393523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7393531Z 
2023-01-11T21:18:21.7393602Z Running tests...
2023-01-11T21:18:21.7393795Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7394066Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23421
2023-01-11T21:18:21.7394223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23422
2023-01-11T21:18:21.7394377Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23423
2023-01-11T21:18:21.7394647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7394768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7395035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7395220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7395467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7395742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7395865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7396147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7396281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7396459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7396716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7396840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7397124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7397259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7397435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7397739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7398040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7398335Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7398501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7398655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7398818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7398923Z skip: Need at least 4 CUDA devices (2.189s)
2023-01-11T21:18:21.7398929Z 
2023-01-11T21:18:21.7399126Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7399203Z Ran 1 test in 2.189s
2023-01-11T21:18:21.7399208Z 
2023-01-11T21:18:21.7399282Z OK (skipped=1)
2023-01-11T21:18:21.7399287Z 
2023-01-11T21:18:21.7399374Z Generating XML reports...
2023-01-11T21:18:21.7399716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211533.xml
2023-01-11T21:18:21.7399951Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7400209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7400335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7400620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7400755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7400761Z 
2023-01-11T21:18:21.7400832Z Running tests...
2023-01-11T21:18:21.7401028Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7401309Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23474
2023-01-11T21:18:21.7401466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23475
2023-01-11T21:18:21.7401703Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23476
2023-01-11T21:18:21.7402008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7402130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7402438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7402574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7402755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7403029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7403153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7403434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7403554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7403738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7404012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7404135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7404412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7404547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7404724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7405023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7405321Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7405604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7405770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7405931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7406093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7406196Z skip: Need at least 4 CUDA devices (2.171s)
2023-01-11T21:18:21.7406202Z 
2023-01-11T21:18:21.7406399Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7406473Z Ran 1 test in 2.171s
2023-01-11T21:18:21.7406479Z 
2023-01-11T21:18:21.7406550Z OK (skipped=1)
2023-01-11T21:18:21.7406556Z 
2023-01-11T21:18:21.7406638Z Generating XML reports...
2023-01-11T21:18:21.7406967Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211537.xml
2023-01-11T21:18:21.7407198Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7407471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7407593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7407871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7408005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7408010Z 
2023-01-11T21:18:21.7408083Z Running tests...
2023-01-11T21:18:21.7408275Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7409012Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.258s)
2023-01-11T21:18:21.7409046Z 
2023-01-11T21:18:21.7409243Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7409304Z Ran 1 test in 0.259s
2023-01-11T21:18:21.7409309Z 
2023-01-11T21:18:21.7409378Z OK (skipped=1)
2023-01-11T21:18:21.7409383Z 
2023-01-11T21:18:21.7409466Z Generating XML reports...
2023-01-11T21:18:21.7409806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211541.xml
2023-01-11T21:18:21.7410038Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7410309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7410434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7410715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7410837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7410854Z 
2023-01-11T21:18:21.7410914Z Running tests...
2023-01-11T21:18:21.7411111Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7411347Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23537
2023-01-11T21:18:21.7411503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23538
2023-01-11T21:18:21.7411657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23539
2023-01-11T21:18:21.7411928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7412051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7412333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7412454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7412630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7412901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7413022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7413300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7413433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7413615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7413887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7414007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7414272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7414404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7414580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7414879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7415177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7415497Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7415691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7415854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7416018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7416181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7416353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7416524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7416818Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7417117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7417406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7417655Z STAGE:2023-01-11 21:15:45 23539:23539 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7417898Z STAGE:2023-01-11 21:15:45 23538:23538 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7418134Z STAGE:2023-01-11 21:15:45 23537:23537 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7418370Z STAGE:2023-01-11 21:15:45 23539:23539 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7418610Z STAGE:2023-01-11 21:15:45 23538:23538 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7418860Z STAGE:2023-01-11 21:15:45 23537:23537 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7419117Z STAGE:2023-01-11 21:15:45 23539:23539 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7419373Z STAGE:2023-01-11 21:15:45 23538:23538 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7419626Z STAGE:2023-01-11 21:15:45 23537:23537 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7419869Z STAGE:2023-01-11 21:15:45 23539:23539 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7420106Z STAGE:2023-01-11 21:15:45 23538:23538 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7420341Z STAGE:2023-01-11 21:15:45 23537:23537 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7420573Z STAGE:2023-01-11 21:15:45 23539:23539 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7420829Z STAGE:2023-01-11 21:15:45 23539:23539 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7421074Z STAGE:2023-01-11 21:15:45 23537:23537 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7421321Z STAGE:2023-01-11 21:15:45 23538:23538 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7421576Z STAGE:2023-01-11 21:15:45 23537:23537 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7421948Z STAGE:2023-01-11 21:15:45 23538:23538 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7422192Z STAGE:2023-01-11 21:15:45 23539:23539 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7422427Z STAGE:2023-01-11 21:15:45 23537:23537 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7422669Z STAGE:2023-01-11 21:15:45 23538:23538 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7423153Z STAGE:2023-01-11 21:15:45 23538:23538 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:15:45 23537:23537 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7423172Z 
2023-01-11T21:18:21.7423408Z STAGE:2023-01-11 21:15:45 23539:23539 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7423842Z STAGE:2023-01-11 21:15:45 23537:23537 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:15:45 23538:23538 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7423848Z 
2023-01-11T21:18:21.7424104Z STAGE:2023-01-11 21:15:45 23539:23539 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7424174Z ok (2.272s)
2023-01-11T21:18:21.7424180Z 
2023-01-11T21:18:21.7424379Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7424459Z Ran 1 test in 2.272s
2023-01-11T21:18:21.7424465Z 
2023-01-11T21:18:21.7424526Z OK
2023-01-11T21:18:21.7424531Z 
2023-01-11T21:18:21.7424617Z Generating XML reports...
2023-01-11T21:18:21.7424964Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211543.xml
2023-01-11T21:18:21.7425184Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7425464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7425588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7425871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7426008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7426014Z 
2023-01-11T21:18:21.7426090Z Running tests...
2023-01-11T21:18:21.7426287Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7426527Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23602
2023-01-11T21:18:21.7426673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23603
2023-01-11T21:18:21.7426832Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23604
2023-01-11T21:18:21.7427111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7427235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7427521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7427657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7427839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7428117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7428242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7428513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7428649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7428829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7429130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7429403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7429559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7429866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7430003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7430180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7430462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7430763Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7430930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7431093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7431259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7431438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7431611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7431784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7432082Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7432362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7432661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7432913Z STAGE:2023-01-11 21:15:49 23604:23604 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7433321Z STAGE:2023-01-11 21:15:49 23602:23602 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:15:49 23603:23603 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7433328Z 
2023-01-11T21:18:21.7433577Z STAGE:2023-01-11 21:15:49 23604:23604 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7433825Z STAGE:2023-01-11 21:15:49 23603:23603 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7434083Z STAGE:2023-01-11 21:15:49 23604:23604 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7434327Z STAGE:2023-01-11 21:15:49 23602:23602 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7434582Z STAGE:2023-01-11 21:15:49 23603:23603 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7434833Z STAGE:2023-01-11 21:15:49 23602:23602 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7435066Z STAGE:2023-01-11 21:15:49 23604:23604 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7435307Z STAGE:2023-01-11 21:15:49 23603:23603 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7435542Z STAGE:2023-01-11 21:15:49 23602:23602 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7435787Z STAGE:2023-01-11 21:15:49 23604:23604 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7436041Z STAGE:2023-01-11 21:15:49 23604:23604 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7436284Z STAGE:2023-01-11 21:15:49 23602:23602 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7436537Z STAGE:2023-01-11 21:15:49 23602:23602 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7436813Z STAGE:2023-01-11 21:15:49 23603:23603 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7437095Z STAGE:2023-01-11 21:15:49 23603:23603 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7437324Z STAGE:2023-01-11 21:15:49 23604:23604 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7437564Z STAGE:2023-01-11 21:15:49 23602:23602 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7437800Z STAGE:2023-01-11 21:15:49 23603:23603 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7438045Z STAGE:2023-01-11 21:15:49 23602:23602 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7438466Z STAGE:2023-01-11 21:15:49 23602:23602 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:15:49 23603:23603 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7438475Z 
2023-01-11T21:18:21.7438721Z STAGE:2023-01-11 21:15:49 23604:23604 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7438979Z STAGE:2023-01-11 21:15:49 23603:23603 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7439233Z STAGE:2023-01-11 21:15:49 23604:23604 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7439304Z ok (2.273s)
2023-01-11T21:18:21.7439309Z 
2023-01-11T21:18:21.7439496Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7439570Z Ran 1 test in 2.273s
2023-01-11T21:18:21.7439575Z 
2023-01-11T21:18:21.7439636Z OK
2023-01-11T21:18:21.7439642Z 
2023-01-11T21:18:21.7439725Z Generating XML reports...
2023-01-11T21:18:21.7440073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211547.xml
2023-01-11T21:18:21.7440307Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7440585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7440710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7440993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7441117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7441122Z 
2023-01-11T21:18:21.7441193Z Running tests...
2023-01-11T21:18:21.7441388Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7441701Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23667
2023-01-11T21:18:21.7441862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23668
2023-01-11T21:18:21.7442023Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23669
2023-01-11T21:18:21.7442301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7442427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7442697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7442832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7443010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7443284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7443407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7443687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7443871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7444075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7444345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7444456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7444735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7444868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7445044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7445345Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7445646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7445945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7446109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7446272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7446422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7446596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7446768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7446940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7447240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7447537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7447830Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7448079Z STAGE:2023-01-11 21:15:54 23668:23668 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7448485Z STAGE:2023-01-11 21:15:54 23669:23669 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:15:54 23667:23667 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7448492Z 
2023-01-11T21:18:21.7448739Z STAGE:2023-01-11 21:15:54 23669:23669 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7448977Z STAGE:2023-01-11 21:15:54 23668:23668 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7449220Z STAGE:2023-01-11 21:15:54 23667:23667 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7449476Z STAGE:2023-01-11 21:15:54 23669:23669 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7449733Z STAGE:2023-01-11 21:15:54 23668:23668 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7449983Z STAGE:2023-01-11 21:15:54 23667:23667 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7450381Z STAGE:2023-01-11 21:15:54 23668:23668 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:15:54 23669:23669 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7450387Z 
2023-01-11T21:18:21.7450627Z STAGE:2023-01-11 21:15:54 23667:23667 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7450902Z STAGE:2023-01-11 21:15:54 23669:23669 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7451185Z STAGE:2023-01-11 21:15:54 23669:23669 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7451434Z STAGE:2023-01-11 21:15:54 23667:23667 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7451664Z STAGE:2023-01-11 21:15:54 23668:23668 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7451917Z STAGE:2023-01-11 21:15:54 23667:23667 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7452170Z STAGE:2023-01-11 21:15:54 23668:23668 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7452411Z STAGE:2023-01-11 21:15:54 23669:23669 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7452650Z STAGE:2023-01-11 21:15:54 23667:23667 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7452887Z STAGE:2023-01-11 21:15:54 23668:23668 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7453302Z STAGE:2023-01-11 21:15:54 23667:23667 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:15:54 23669:23669 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7453308Z 
2023-01-11T21:18:21.7453563Z STAGE:2023-01-11 21:15:54 23669:23669 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7453816Z STAGE:2023-01-11 21:15:54 23667:23667 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7454059Z STAGE:2023-01-11 21:15:54 23668:23668 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7454301Z STAGE:2023-01-11 21:15:54 23668:23668 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7454371Z ok (2.375s)
2023-01-11T21:18:21.7454377Z 
2023-01-11T21:18:21.7454574Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7454649Z Ran 1 test in 2.375s
2023-01-11T21:18:21.7454657Z 
2023-01-11T21:18:21.7454720Z OK
2023-01-11T21:18:21.7454725Z 
2023-01-11T21:18:21.7454810Z Generating XML reports...
2023-01-11T21:18:21.7455154Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211552.xml
2023-01-11T21:18:21.7455384Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7455649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7455772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7456053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7456190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7456195Z 
2023-01-11T21:18:21.7456268Z Running tests...
2023-01-11T21:18:21.7456467Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7456703Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23732
2023-01-11T21:18:21.7456862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23733
2023-01-11T21:18:21.7457016Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23734
2023-01-11T21:18:21.7457275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7457398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7457676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7457841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7458045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7458319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7458441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7458719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7458841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7459015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7459285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7459409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7459691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7459823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7459997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7460298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7460596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7460876Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7461041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7461206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7461366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7461539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7461825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7462000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7462294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7462590Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7462826Z STAGE:2023-01-11 21:15:58 23733:23733 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7463077Z STAGE:2023-01-11 21:15:58 23734:23734 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7463327Z STAGE:2023-01-11 21:15:58 23734:23734 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7463584Z STAGE:2023-01-11 21:15:58 23734:23734 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7463827Z STAGE:2023-01-11 21:15:58 23734:23734 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7464126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7464370Z STAGE:2023-01-11 21:15:58 23732:23732 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7464618Z STAGE:2023-01-11 21:15:58 23732:23732 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7464861Z STAGE:2023-01-11 21:15:58 23733:23733 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7465195Z STAGE:2023-01-11 21:15:58 23732:23732 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7465453Z STAGE:2023-01-11 21:15:58 23733:23733 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7465699Z STAGE:2023-01-11 21:15:58 23733:23733 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7465939Z STAGE:2023-01-11 21:15:58 23732:23732 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7466186Z STAGE:2023-01-11 21:15:58 23734:23734 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7466611Z STAGE:2023-01-11 21:15:58 23734:23734 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:15:58 23732:23732 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7466617Z 
2023-01-11T21:18:21.7466869Z STAGE:2023-01-11 21:15:58 23733:23733 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7467128Z STAGE:2023-01-11 21:15:58 23732:23732 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7467384Z STAGE:2023-01-11 21:15:58 23733:23733 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7467626Z STAGE:2023-01-11 21:15:58 23734:23734 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7467851Z STAGE:2023-01-11 21:15:58 23732:23732 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7468094Z STAGE:2023-01-11 21:15:58 23733:23733 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7468341Z STAGE:2023-01-11 21:15:58 23732:23732 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7468583Z STAGE:2023-01-11 21:15:58 23733:23733 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7468843Z STAGE:2023-01-11 21:15:58 23732:23732 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7469090Z STAGE:2023-01-11 21:15:58 23734:23734 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7469347Z STAGE:2023-01-11 21:15:58 23733:23733 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7469606Z STAGE:2023-01-11 21:15:58 23734:23734 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7469674Z ok (2.283s)
2023-01-11T21:18:21.7469679Z 
2023-01-11T21:18:21.7469865Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7469940Z Ran 1 test in 2.283s
2023-01-11T21:18:21.7469946Z 
2023-01-11T21:18:21.7470005Z OK
2023-01-11T21:18:21.7470010Z 
2023-01-11T21:18:21.7470093Z Generating XML reports...
2023-01-11T21:18:21.7470435Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211556.xml
2023-01-11T21:18:21.7470667Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7470947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7471071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7471340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7471474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7471480Z 
2023-01-11T21:18:21.7471552Z Running tests...
2023-01-11T21:18:21.7471749Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7471980Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23797
2023-01-11T21:18:21.7472171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23798
2023-01-11T21:18:21.7472328Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23799
2023-01-11T21:18:21.7472630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7472754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7473021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7473154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7473329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7473602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7473723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7474007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7474144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7474320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7474618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7474875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7474997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7475272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7475405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7475580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7475879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7476176Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7476339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7476491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7476651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7476821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7476994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7477168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7477467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7477760Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7478169Z STAGE:2023-01-11 21:16:02 23798:23798 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:16:02 23799:23799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7478176Z 
2023-01-11T21:18:21.7478585Z STAGE:2023-01-11 21:16:02 23798:23798 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:16:02 23799:23799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7478591Z 
2023-01-11T21:18:21.7479080Z STAGE:2023-01-11 21:16:02 23798:23798 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:16:02 23799:23799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7479087Z 
2023-01-11T21:18:21.7479333Z STAGE:2023-01-11 21:16:02 23799:23799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7479571Z STAGE:2023-01-11 21:16:02 23798:23798 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7479807Z STAGE:2023-01-11 21:16:02 23799:23799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7480050Z STAGE:2023-01-11 21:16:02 23798:23798 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7480477Z STAGE:2023-01-11 21:16:02 23799:23799 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:16:02 23798:23798 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7480485Z 
2023-01-11T21:18:21.7480785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7480857Z ok (2.275s)
2023-01-11T21:18:21.7480864Z 
2023-01-11T21:18:21.7481064Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7481139Z Ran 1 test in 2.275s
2023-01-11T21:18:21.7481144Z 
2023-01-11T21:18:21.7481204Z OK
2023-01-11T21:18:21.7481208Z 
2023-01-11T21:18:21.7481292Z Generating XML reports...
2023-01-11T21:18:21.7481692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211600.xml
2023-01-11T21:18:21.7481930Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7482206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7482333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7482620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7482761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7482767Z 
2023-01-11T21:18:21.7482841Z Running tests...
2023-01-11T21:18:21.7483040Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7483271Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23858
2023-01-11T21:18:21.7483417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23859
2023-01-11T21:18:21.7483575Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23860
2023-01-11T21:18:21.7483849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7483975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7484260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7484396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7484572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7484840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7484963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7485231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7485366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7485543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7485849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7486002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7486284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7486416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7486591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7486877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7487180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7487476Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7487642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7487811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7487974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7488155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7488332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7488505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7488788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7489085Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7489499Z STAGE:2023-01-11 21:16:06 23860:23860 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:16:06 23859:23859 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7489506Z 
2023-01-11T21:18:21.7489756Z STAGE:2023-01-11 21:16:06 23860:23860 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7490001Z STAGE:2023-01-11 21:16:06 23859:23859 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7490259Z STAGE:2023-01-11 21:16:06 23860:23860 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7490515Z STAGE:2023-01-11 21:16:06 23859:23859 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7490762Z STAGE:2023-01-11 21:16:06 23860:23860 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7491003Z STAGE:2023-01-11 21:16:06 23859:23859 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7491253Z STAGE:2023-01-11 21:16:06 23859:23859 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7491482Z STAGE:2023-01-11 21:16:06 23860:23860 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7491743Z STAGE:2023-01-11 21:16:06 23859:23859 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7491996Z STAGE:2023-01-11 21:16:06 23860:23860 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7492297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7492367Z ok (2.282s)
2023-01-11T21:18:21.7492372Z 
2023-01-11T21:18:21.7492570Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7492687Z Ran 1 test in 2.282s
2023-01-11T21:18:21.7492693Z 
2023-01-11T21:18:21.7492756Z OK
2023-01-11T21:18:21.7492761Z 
2023-01-11T21:18:21.7492847Z Generating XML reports...
2023-01-11T21:18:21.7493206Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211604.xml
2023-01-11T21:18:21.7493444Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7493722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7493847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7494128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7494264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7494270Z 
2023-01-11T21:18:21.7494343Z Running tests...
2023-01-11T21:18:21.7494547Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7494774Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23919
2023-01-11T21:18:21.7494933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23920
2023-01-11T21:18:21.7495094Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23921
2023-01-11T21:18:21.7495370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7495495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7495782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7495918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7496095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7496369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7496480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7496762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7496898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7497072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7497339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7497461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7497741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7497876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7498053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7498421Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7498765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7499056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7499220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7499396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7499593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7499755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7499953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7500126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7500407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7500697Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7500947Z STAGE:2023-01-11 21:16:10 23920:23920 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7501190Z STAGE:2023-01-11 21:16:10 23921:23921 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7501438Z STAGE:2023-01-11 21:16:10 23921:23921 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7501786Z STAGE:2023-01-11 21:16:10 23920:23920 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7502049Z STAGE:2023-01-11 21:16:10 23921:23921 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7502305Z STAGE:2023-01-11 21:16:10 23920:23920 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7502552Z STAGE:2023-01-11 21:16:10 23921:23921 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7502778Z STAGE:2023-01-11 21:16:10 23920:23920 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7503024Z STAGE:2023-01-11 21:16:10 23920:23920 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7503266Z STAGE:2023-01-11 21:16:10 23921:23921 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7503526Z STAGE:2023-01-11 21:16:10 23920:23920 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7503781Z STAGE:2023-01-11 21:16:10 23921:23921 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7504080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7504150Z ok (2.378s)
2023-01-11T21:18:21.7504156Z 
2023-01-11T21:18:21.7504359Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7504436Z Ran 1 test in 2.378s
2023-01-11T21:18:21.7504442Z 
2023-01-11T21:18:21.7504490Z OK
2023-01-11T21:18:21.7504494Z 
2023-01-11T21:18:21.7504579Z Generating XML reports...
2023-01-11T21:18:21.7504923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211608.xml
2023-01-11T21:18:21.7505156Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7505436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7505562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7505845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7505981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7505987Z 
2023-01-11T21:18:21.7506047Z Running tests...
2023-01-11T21:18:21.7506241Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7506470Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23980
2023-01-11T21:18:21.7506625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23981
2023-01-11T21:18:21.7506868Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23982
2023-01-11T21:18:21.7507179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7507304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7507587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7507724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7507887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7508159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7508285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7508567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7508704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7508887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7509159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7509282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7509548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7509680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7509856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7510154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7510449Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7510746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7510910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7511073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7511234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7511395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7511567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7511736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7512033Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7512327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7512574Z STAGE:2023-01-11 21:16:14 23982:23982 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7512813Z STAGE:2023-01-11 21:16:14 23981:23981 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7513062Z STAGE:2023-01-11 21:16:14 23982:23982 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7513303Z STAGE:2023-01-11 21:16:14 23981:23981 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7513548Z STAGE:2023-01-11 21:16:14 23982:23982 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7513831Z STAGE:2023-01-11 21:16:14 23981:23981 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7514100Z STAGE:2023-01-11 21:16:14 23982:23982 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7514340Z STAGE:2023-01-11 21:16:14 23981:23981 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7514754Z STAGE:2023-01-11 21:16:14 23982:23982 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:16:14 23981:23981 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7514761Z 
2023-01-11T21:18:21.7515188Z STAGE:2023-01-11 21:16:14 23981:23981 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:16:14 23982:23982 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7515195Z 
2023-01-11T21:18:21.7515491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7515561Z ok (2.380s)
2023-01-11T21:18:21.7515566Z 
2023-01-11T21:18:21.7515767Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7515841Z Ran 1 test in 2.380s
2023-01-11T21:18:21.7515847Z 
2023-01-11T21:18:21.7515906Z OK
2023-01-11T21:18:21.7515912Z 
2023-01-11T21:18:21.7515982Z Generating XML reports...
2023-01-11T21:18:21.7516326Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211612.xml
2023-01-11T21:18:21.7516557Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7516833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7516956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7517237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7517375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7517381Z 
2023-01-11T21:18:21.7517457Z Running tests...
2023-01-11T21:18:21.7517642Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7517871Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24041
2023-01-11T21:18:21.7518026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24042
2023-01-11T21:18:21.7518179Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24043
2023-01-11T21:18:21.7518454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7518576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7518852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7530181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7530442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7530773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7530901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7531192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7531330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7531498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7531805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7532231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7532356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7532640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7532776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7532952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7533254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7533556Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7533712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7533876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7534042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7534296Z STAGE:2023-01-11 21:16:18 24041:24041 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7534539Z STAGE:2023-01-11 21:16:18 24042:24042 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7534782Z STAGE:2023-01-11 21:16:18 24043:24043 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7535034Z STAGE:2023-01-11 21:16:18 24043:24043 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7535279Z STAGE:2023-01-11 21:16:18 24042:24042 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7535539Z STAGE:2023-01-11 21:16:18 24043:24043 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7535778Z STAGE:2023-01-11 21:16:18 24041:24041 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7536039Z STAGE:2023-01-11 21:16:18 24042:24042 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7536297Z STAGE:2023-01-11 21:16:18 24041:24041 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7536536Z STAGE:2023-01-11 21:16:18 24042:24042 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7536774Z STAGE:2023-01-11 21:16:18 24043:24043 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7537011Z STAGE:2023-01-11 21:16:18 24041:24041 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7537259Z STAGE:2023-01-11 21:16:18 24043:24043 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7537516Z STAGE:2023-01-11 21:16:18 24043:24043 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7537764Z STAGE:2023-01-11 21:16:18 24041:24041 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7537994Z STAGE:2023-01-11 21:16:18 24042:24042 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7538250Z STAGE:2023-01-11 21:16:18 24041:24041 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7538501Z STAGE:2023-01-11 21:16:18 24042:24042 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7538739Z STAGE:2023-01-11 21:16:18 24043:24043 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7538977Z STAGE:2023-01-11 21:16:18 24041:24041 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7539213Z STAGE:2023-01-11 21:16:18 24042:24042 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7539687Z STAGE:2023-01-11 21:16:18 24041:24041 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:16:18 24042:24042 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7539697Z 
2023-01-11T21:18:21.7539943Z STAGE:2023-01-11 21:16:18 24043:24043 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7540375Z STAGE:2023-01-11 21:16:18 24041:24041 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:16:18 24042:24042 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7540382Z 
2023-01-11T21:18:21.7540638Z STAGE:2023-01-11 21:16:18 24043:24043 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7540696Z ok (2.279s)
2023-01-11T21:18:21.7540715Z 
2023-01-11T21:18:21.7540901Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7540984Z Ran 1 test in 2.280s
2023-01-11T21:18:21.7540989Z 
2023-01-11T21:18:21.7541051Z OK
2023-01-11T21:18:21.7541056Z 
2023-01-11T21:18:21.7541142Z Generating XML reports...
2023-01-11T21:18:21.7541489Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211617.xml
2023-01-11T21:18:21.7541906Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7542187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7542313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7542585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7542724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7542729Z 
2023-01-11T21:18:21.7542802Z Running tests...
2023-01-11T21:18:21.7543006Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7543237Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24097
2023-01-11T21:18:21.7543397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24098
2023-01-11T21:18:21.7543552Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24099
2023-01-11T21:18:21.7543826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7543955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7544225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7544363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7544541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7544820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7544947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7545233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7545368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7545547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7545807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7545933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7546215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7546416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7546628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7546938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7547240Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7547533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7547701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7547856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7548017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7548271Z STAGE:2023-01-11 21:16:22 24098:24098 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7548518Z STAGE:2023-01-11 21:16:22 24099:24099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7548763Z STAGE:2023-01-11 21:16:22 24097:24097 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7549008Z STAGE:2023-01-11 21:16:22 24099:24099 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7549266Z STAGE:2023-01-11 21:16:22 24099:24099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7549514Z STAGE:2023-01-11 21:16:22 24098:24098 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7549787Z STAGE:2023-01-11 21:16:22 24097:24097 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7550113Z STAGE:2023-01-11 21:16:22 24098:24098 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7550376Z STAGE:2023-01-11 21:16:22 24097:24097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7550619Z STAGE:2023-01-11 21:16:22 24099:24099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7550858Z STAGE:2023-01-11 21:16:22 24098:24098 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7551095Z STAGE:2023-01-11 21:16:22 24097:24097 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7551343Z STAGE:2023-01-11 21:16:22 24099:24099 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7551597Z STAGE:2023-01-11 21:16:22 24099:24099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7551841Z STAGE:2023-01-11 21:16:22 24097:24097 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7552086Z STAGE:2023-01-11 21:16:22 24098:24098 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7552331Z STAGE:2023-01-11 21:16:22 24097:24097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7552588Z STAGE:2023-01-11 21:16:22 24098:24098 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7552827Z STAGE:2023-01-11 21:16:22 24099:24099 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7553065Z STAGE:2023-01-11 21:16:22 24097:24097 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7553299Z STAGE:2023-01-11 21:16:22 24098:24098 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7553545Z STAGE:2023-01-11 21:16:22 24097:24097 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7553793Z STAGE:2023-01-11 21:16:22 24098:24098 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7554090Z STAGE:2023-01-11 21:16:22 24099:24099 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7554373Z STAGE:2023-01-11 21:16:22 24097:24097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7554616Z STAGE:2023-01-11 21:16:22 24098:24098 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7554867Z STAGE:2023-01-11 21:16:22 24099:24099 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7554936Z ok (2.276s)
2023-01-11T21:18:21.7554942Z 
2023-01-11T21:18:21.7555144Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7555220Z Ran 1 test in 2.276s
2023-01-11T21:18:21.7555226Z 
2023-01-11T21:18:21.7555289Z OK
2023-01-11T21:18:21.7555294Z 
2023-01-11T21:18:21.7555382Z Generating XML reports...
2023-01-11T21:18:21.7555733Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211621.xml
2023-01-11T21:18:21.7555958Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7556236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7556362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7556649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7556786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7556792Z 
2023-01-11T21:18:21.7556867Z Running tests...
2023-01-11T21:18:21.7557070Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7557273Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2023-01-11T21:18:21.7557279Z 
2023-01-11T21:18:21.7557478Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7557542Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7557561Z 
2023-01-11T21:18:21.7557622Z OK (skipped=1)
2023-01-11T21:18:21.7557629Z 
2023-01-11T21:18:21.7557715Z Generating XML reports...
2023-01-11T21:18:21.7558054Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211625.xml
2023-01-11T21:18:21.7558288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7558567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7558694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7558976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7559116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7559124Z 
2023-01-11T21:18:21.7559186Z Running tests...
2023-01-11T21:18:21.7559385Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7559619Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24163
2023-01-11T21:18:21.7559780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24164
2023-01-11T21:18:21.7559939Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24165
2023-01-11T21:18:21.7560213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7560337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7560617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7560768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7560948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7561249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7561374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7561723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7561863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7562043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7562315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7562437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7562705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7562845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7563022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7563326Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7563624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7563917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7564084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7564249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7564415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7564654Z STAGE:2023-01-11 21:16:28 24165:24165 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7564899Z STAGE:2023-01-11 21:16:28 24163:24163 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7565144Z STAGE:2023-01-11 21:16:28 24164:24164 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7565391Z STAGE:2023-01-11 21:16:28 24165:24165 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7565637Z STAGE:2023-01-11 21:16:28 24164:24164 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7565878Z STAGE:2023-01-11 21:16:28 24163:24163 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7566136Z STAGE:2023-01-11 21:16:28 24165:24165 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7566398Z STAGE:2023-01-11 21:16:28 24164:24164 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7566658Z STAGE:2023-01-11 21:16:28 24163:24163 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7566885Z STAGE:2023-01-11 21:16:28 24165:24165 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7567125Z STAGE:2023-01-11 21:16:28 24164:24164 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7567362Z STAGE:2023-01-11 21:16:28 24163:24163 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7567609Z STAGE:2023-01-11 21:16:28 24165:24165 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7567864Z STAGE:2023-01-11 21:16:28 24165:24165 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7568313Z STAGE:2023-01-11 21:16:28 24163:24163 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:16:28 24164:24164 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7568346Z 
2023-01-11T21:18:21.7568781Z STAGE:2023-01-11 21:16:28 24164:24164 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:16:28 24163:24163 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7568788Z 
2023-01-11T21:18:21.7569033Z STAGE:2023-01-11 21:16:28 24165:24165 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7569273Z STAGE:2023-01-11 21:16:28 24163:24163 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7569510Z STAGE:2023-01-11 21:16:28 24164:24164 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7569744Z STAGE:2023-01-11 21:16:28 24165:24165 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7569993Z STAGE:2023-01-11 21:16:28 24164:24164 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7570244Z STAGE:2023-01-11 21:16:28 24163:24163 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7570501Z STAGE:2023-01-11 21:16:28 24165:24165 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7570935Z STAGE:2023-01-11 21:16:28 24164:24164 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:16:28 24163:24163 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7570941Z 
2023-01-11T21:18:21.7571010Z ok (2.277s)
2023-01-11T21:18:21.7571016Z 
2023-01-11T21:18:21.7571216Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7571293Z Ran 1 test in 2.278s
2023-01-11T21:18:21.7571298Z 
2023-01-11T21:18:21.7571360Z OK
2023-01-11T21:18:21.7571365Z 
2023-01-11T21:18:21.7571454Z Generating XML reports...
2023-01-11T21:18:21.7571785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211626.xml
2023-01-11T21:18:21.7572023Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7572299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7572426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7572712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7572851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7572856Z 
2023-01-11T21:18:21.7572931Z Running tests...
2023-01-11T21:18:21.7573130Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7573334Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2023-01-11T21:18:21.7573355Z 
2023-01-11T21:18:21.7573542Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7573619Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7573624Z 
2023-01-11T21:18:21.7573697Z OK (skipped=1)
2023-01-11T21:18:21.7573703Z 
2023-01-11T21:18:21.7573789Z Generating XML reports...
2023-01-11T21:18:21.7574128Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211631.xml
2023-01-11T21:18:21.7574360Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7574633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7574757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7575027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7575195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7575199Z 
2023-01-11T21:18:21.7575301Z Running tests...
2023-01-11T21:18:21.7575501Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7575702Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.002s)
2023-01-11T21:18:21.7575707Z 
2023-01-11T21:18:21.7575904Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7575980Z Ran 1 test in 0.003s
2023-01-11T21:18:21.7575986Z 
2023-01-11T21:18:21.7576058Z OK (skipped=1)
2023-01-11T21:18:21.7576063Z 
2023-01-11T21:18:21.7576148Z Generating XML reports...
2023-01-11T21:18:21.7576472Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211632.xml
2023-01-11T21:18:21.7576708Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7576980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7577104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7577386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7577521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7577527Z 
2023-01-11T21:18:21.7577601Z Running tests...
2023-01-11T21:18:21.7577795Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7578021Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24239
2023-01-11T21:18:21.7578166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24240
2023-01-11T21:18:21.7578326Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24241
2023-01-11T21:18:21.7578603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7578730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7579013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7579150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7579330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7579604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7579713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7579997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7580134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7580314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7580614Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7580884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7581007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7581288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7581422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7581584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7582095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7582400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7582567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7582729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7582893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7583142Z STAGE:2023-01-11 21:16:36 24241:24241 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7583387Z STAGE:2023-01-11 21:16:36 24239:24239 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7583627Z STAGE:2023-01-11 21:16:36 24240:24240 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7583866Z STAGE:2023-01-11 21:16:36 24240:24240 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7584114Z STAGE:2023-01-11 21:16:36 24241:24241 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7584355Z STAGE:2023-01-11 21:16:36 24239:24239 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7584616Z STAGE:2023-01-11 21:16:36 24240:24240 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7584876Z STAGE:2023-01-11 21:16:36 24241:24241 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7585134Z STAGE:2023-01-11 21:16:36 24239:24239 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7585532Z STAGE:2023-01-11 21:16:36 24240:24240 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:16:36 24241:24241 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7585540Z 
2023-01-11T21:18:21.7585783Z STAGE:2023-01-11 21:16:36 24239:24239 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7586036Z STAGE:2023-01-11 21:16:36 24241:24241 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7586292Z STAGE:2023-01-11 21:16:36 24241:24241 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7586526Z STAGE:2023-01-11 21:16:36 24239:24239 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7586773Z STAGE:2023-01-11 21:16:36 24240:24240 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7587032Z STAGE:2023-01-11 21:16:36 24239:24239 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7587285Z STAGE:2023-01-11 21:16:36 24240:24240 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7587528Z STAGE:2023-01-11 21:16:36 24241:24241 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7587765Z STAGE:2023-01-11 21:16:36 24239:24239 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7588000Z STAGE:2023-01-11 21:16:36 24240:24240 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7588244Z STAGE:2023-01-11 21:16:36 24239:24239 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7588485Z STAGE:2023-01-11 21:16:36 24240:24240 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7588895Z STAGE:2023-01-11 21:16:36 24241:24241 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:16:36 24239:24239 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7588915Z 
2023-01-11T21:18:21.7589158Z STAGE:2023-01-11 21:16:36 24240:24240 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7589454Z STAGE:2023-01-11 21:16:36 24241:24241 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7589550Z ok (2.278s)
2023-01-11T21:18:21.7589556Z 
2023-01-11T21:18:21.7589758Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7589835Z Ran 1 test in 2.278s
2023-01-11T21:18:21.7589840Z 
2023-01-11T21:18:21.7589901Z OK
2023-01-11T21:18:21.7589906Z 
2023-01-11T21:18:21.7589993Z Generating XML reports...
2023-01-11T21:18:21.7590336Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211634.xml
2023-01-11T21:18:21.7590559Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7590840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7590968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7591254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7591393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7591398Z 
2023-01-11T21:18:21.7591473Z Running tests...
2023-01-11T21:18:21.7591671Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7591858Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2023-01-11T21:18:21.7591863Z 
2023-01-11T21:18:21.7592059Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7592123Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7592128Z 
2023-01-11T21:18:21.7592200Z OK (skipped=1)
2023-01-11T21:18:21.7592205Z 
2023-01-11T21:18:21.7592289Z Generating XML reports...
2023-01-11T21:18:21.7592631Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211638.xml
2023-01-11T21:18:21.7592871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7593148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7593272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7593554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7593677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7593698Z 
2023-01-11T21:18:21.7593757Z Running tests...
2023-01-11T21:18:21.7593952Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7594146Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2023-01-11T21:18:21.7594154Z 
2023-01-11T21:18:21.7594349Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7594426Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7594432Z 
2023-01-11T21:18:21.7594507Z OK (skipped=1)
2023-01-11T21:18:21.7594512Z 
2023-01-11T21:18:21.7594596Z Generating XML reports...
2023-01-11T21:18:21.7594937Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211640.xml
2023-01-11T21:18:21.7595155Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7595434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7595556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7595837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7596009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7596015Z 
2023-01-11T21:18:21.7596088Z Running tests...
2023-01-11T21:18:21.7596313Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7596547Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24315
2023-01-11T21:18:21.7596704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24316
2023-01-11T21:18:21.7596846Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24317
2023-01-11T21:18:21.7597121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7597247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7597528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7597666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7597849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7598124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7598247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7598513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7598651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7598827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7599101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7599230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7599508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7599645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7599819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7600119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7600407Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7600702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7600869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7601036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7601201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7601450Z STAGE:2023-01-11 21:16:43 24317:24317 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7601759Z STAGE:2023-01-11 21:16:43 24315:24315 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7602006Z STAGE:2023-01-11 21:16:43 24316:24316 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7602258Z STAGE:2023-01-11 21:16:43 24317:24317 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7602488Z STAGE:2023-01-11 21:16:43 24316:24316 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7602731Z STAGE:2023-01-11 21:16:43 24315:24315 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7603041Z STAGE:2023-01-11 21:16:43 24317:24317 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7603503Z STAGE:2023-01-11 21:16:43 24316:24316 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:16:43 24315:24315 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7603511Z 
2023-01-11T21:18:21.7603757Z STAGE:2023-01-11 21:16:43 24317:24317 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7603995Z STAGE:2023-01-11 21:16:43 24316:24316 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7604231Z STAGE:2023-01-11 21:16:43 24315:24315 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7604477Z STAGE:2023-01-11 21:16:43 24317:24317 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7604719Z STAGE:2023-01-11 21:16:43 24315:24315 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7604979Z STAGE:2023-01-11 21:16:43 24317:24317 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7605215Z STAGE:2023-01-11 21:16:43 24316:24316 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7605475Z STAGE:2023-01-11 21:16:43 24315:24315 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7605731Z STAGE:2023-01-11 21:16:43 24316:24316 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7606134Z STAGE:2023-01-11 21:16:43 24317:24317 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:16:43 24315:24315 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7606140Z 
2023-01-11T21:18:21.7606380Z STAGE:2023-01-11 21:16:43 24316:24316 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7606628Z STAGE:2023-01-11 21:16:43 24315:24315 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7606873Z STAGE:2023-01-11 21:16:43 24316:24316 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7607118Z STAGE:2023-01-11 21:16:43 24317:24317 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7607375Z STAGE:2023-01-11 21:16:43 24315:24315 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7607631Z STAGE:2023-01-11 21:16:43 24316:24316 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7607871Z STAGE:2023-01-11 21:16:43 24317:24317 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7607940Z ok (2.281s)
2023-01-11T21:18:21.7607945Z 
2023-01-11T21:18:21.7608143Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7608221Z Ran 1 test in 2.282s
2023-01-11T21:18:21.7608226Z 
2023-01-11T21:18:21.7608288Z OK
2023-01-11T21:18:21.7608295Z 
2023-01-11T21:18:21.7608380Z Generating XML reports...
2023-01-11T21:18:21.7608726Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211641.xml
2023-01-11T21:18:21.7608965Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7609226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7609354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7609638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7609775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7609780Z 
2023-01-11T21:18:21.7609853Z Running tests...
2023-01-11T21:18:21.7610052Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7610310Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24371
2023-01-11T21:18:21.7610499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24372
2023-01-11T21:18:21.7610658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24373
2023-01-11T21:18:21.7610919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7611043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7611328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7611465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7611643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7611916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7612042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7612325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7612448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7612627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7612899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7613028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7613310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7613446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7613627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7613931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7614231Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7614514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7614681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7614932Z STAGE:2023-01-11 21:16:47 24371:24371 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7615096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7615259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7615511Z STAGE:2023-01-11 21:16:47 24372:24372 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7615759Z STAGE:2023-01-11 21:16:47 24373:24373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7616006Z STAGE:2023-01-11 21:16:47 24372:24372 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7616252Z STAGE:2023-01-11 21:16:47 24373:24373 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7616480Z STAGE:2023-01-11 21:16:47 24371:24371 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7616738Z STAGE:2023-01-11 21:16:47 24372:24372 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7616994Z STAGE:2023-01-11 21:16:47 24373:24373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7617282Z STAGE:2023-01-11 21:16:47 24371:24371 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7617554Z STAGE:2023-01-11 21:16:47 24372:24372 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7617796Z STAGE:2023-01-11 21:16:47 24373:24373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7618034Z STAGE:2023-01-11 21:16:47 24371:24371 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7618279Z STAGE:2023-01-11 21:16:47 24373:24373 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7618522Z STAGE:2023-01-11 21:16:47 24372:24372 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7618766Z STAGE:2023-01-11 21:16:47 24373:24373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7619016Z STAGE:2023-01-11 21:16:47 24371:24371 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7619275Z STAGE:2023-01-11 21:16:47 24372:24372 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7619536Z STAGE:2023-01-11 21:16:47 24371:24371 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7619776Z STAGE:2023-01-11 21:16:47 24372:24372 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7620178Z STAGE:2023-01-11 21:16:47 24371:24371 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:16:47 24373:24373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7620184Z 
2023-01-11T21:18:21.7620435Z STAGE:2023-01-11 21:16:47 24373:24373 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7620681Z STAGE:2023-01-11 21:16:47 24372:24372 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7620924Z STAGE:2023-01-11 21:16:47 24371:24371 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7621184Z STAGE:2023-01-11 21:16:47 24373:24373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7621429Z STAGE:2023-01-11 21:16:47 24372:24372 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7621801Z STAGE:2023-01-11 21:16:47 24371:24371 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7621873Z ok (2.374s)
2023-01-11T21:18:21.7621879Z 
2023-01-11T21:18:21.7622080Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7622158Z Ran 1 test in 2.374s
2023-01-11T21:18:21.7622164Z 
2023-01-11T21:18:21.7622226Z OK
2023-01-11T21:18:21.7622231Z 
2023-01-11T21:18:21.7622316Z Generating XML reports...
2023-01-11T21:18:21.7622661Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211646.xml
2023-01-11T21:18:21.7622880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7623161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7623289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7623571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7623711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7623715Z 
2023-01-11T21:18:21.7623791Z Running tests...
2023-01-11T21:18:21.7623993Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7624224Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24427
2023-01-11T21:18:21.7624382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24428
2023-01-11T21:18:21.7624575Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24429
2023-01-11T21:18:21.7624918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7625044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7625328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7625467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7625646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7625921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7626046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7626314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7626454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7626633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7626907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7627033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7627315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7627455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7627633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7627938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7628227Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7628525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7628692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7628858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7629023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7629094Z ok (2.172s)
2023-01-11T21:18:21.7629100Z 
2023-01-11T21:18:21.7629300Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7629378Z Ran 1 test in 2.172s
2023-01-11T21:18:21.7629383Z 
2023-01-11T21:18:21.7629446Z OK
2023-01-11T21:18:21.7629451Z 
2023-01-11T21:18:21.7629525Z Generating XML reports...
2023-01-11T21:18:21.7629872Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211650.xml
2023-01-11T21:18:21.7630106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7630385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7630513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7630799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7630935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7630940Z 
2023-01-11T21:18:21.7631015Z Running tests...
2023-01-11T21:18:21.7631200Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7631436Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24480
2023-01-11T21:18:21.7631653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24481
2023-01-11T21:18:21.7631810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24482
2023-01-11T21:18:21.7632085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7632210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7632490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7632626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7632804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7633063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7633190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7633470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7633606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7633785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7634056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7634180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7634461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7634595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7634760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7635062Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7635357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7635649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7635814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7635976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7636140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7636389Z STAGE:2023-01-11 21:16:56 24482:24482 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7636634Z STAGE:2023-01-11 21:16:56 24480:24480 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7636861Z STAGE:2023-01-11 21:16:56 24481:24481 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7637279Z STAGE:2023-01-11 21:16:56 24480:24480 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:16:56 24481:24481 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7637285Z 
2023-01-11T21:18:21.7637535Z STAGE:2023-01-11 21:16:56 24482:24482 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7637966Z STAGE:2023-01-11 21:16:56 24480:24480 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:16:56 24481:24481 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7637972Z 
2023-01-11T21:18:21.7638262Z STAGE:2023-01-11 21:16:56 24482:24482 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7638533Z STAGE:2023-01-11 21:16:56 24482:24482 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7638775Z STAGE:2023-01-11 21:16:56 24481:24481 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7639012Z STAGE:2023-01-11 21:16:56 24480:24480 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7639259Z STAGE:2023-01-11 21:16:56 24480:24480 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7639504Z STAGE:2023-01-11 21:16:56 24482:24482 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7639734Z STAGE:2023-01-11 21:16:56 24481:24481 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7639997Z STAGE:2023-01-11 21:16:56 24480:24480 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7640258Z STAGE:2023-01-11 21:16:56 24482:24482 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7640512Z STAGE:2023-01-11 21:16:56 24481:24481 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7640752Z STAGE:2023-01-11 21:16:56 24480:24480 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7640989Z STAGE:2023-01-11 21:16:56 24482:24482 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7641224Z STAGE:2023-01-11 21:16:56 24481:24481 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7641467Z STAGE:2023-01-11 21:16:56 24480:24480 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7641790Z STAGE:2023-01-11 21:16:56 24480:24480 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7642026Z STAGE:2023-01-11 21:16:56 24481:24481 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7642275Z STAGE:2023-01-11 21:16:56 24482:24482 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7642532Z STAGE:2023-01-11 21:16:56 24481:24481 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7642789Z STAGE:2023-01-11 21:16:56 24482:24482 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7642859Z ok (2.282s)
2023-01-11T21:18:21.7642865Z 
2023-01-11T21:18:21.7643065Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7643141Z Ran 1 test in 2.282s
2023-01-11T21:18:21.7643147Z 
2023-01-11T21:18:21.7643209Z OK
2023-01-11T21:18:21.7643214Z 
2023-01-11T21:18:21.7643299Z Generating XML reports...
2023-01-11T21:18:21.7643630Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211654.xml
2023-01-11T21:18:21.7643866Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7644146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7644275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7644564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7644701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7644706Z 
2023-01-11T21:18:21.7644780Z Running tests...
2023-01-11T21:18:21.7644977Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7645152Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2023-01-11T21:18:21.7645170Z 
2023-01-11T21:18:21.7645353Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7645432Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7645470Z 
2023-01-11T21:18:21.7645544Z OK (skipped=1)
2023-01-11T21:18:21.7645549Z 
2023-01-11T21:18:21.7645634Z Generating XML reports...
2023-01-11T21:18:21.7646004Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211658.xml
2023-01-11T21:18:21.7646242Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7646516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7646641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7646908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7647044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7647049Z 
2023-01-11T21:18:21.7647123Z Running tests...
2023-01-11T21:18:21.7647323Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7647518Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2023-01-11T21:18:21.7647525Z 
2023-01-11T21:18:21.7647721Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7647797Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7647802Z 
2023-01-11T21:18:21.7647872Z OK (skipped=1)
2023-01-11T21:18:21.7647877Z 
2023-01-11T21:18:21.7647963Z Generating XML reports...
2023-01-11T21:18:21.7648291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211700.xml
2023-01-11T21:18:21.7648524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7648800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7648925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7649212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7649350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7649356Z 
2023-01-11T21:18:21.7649431Z Running tests...
2023-01-11T21:18:21.7649628Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7649850Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24556
2023-01-11T21:18:21.7650010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24557
2023-01-11T21:18:21.7650168Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24558
2023-01-11T21:18:21.7650439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7650566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7650851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7650990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7651171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7651444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7651553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7651833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7651969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7652148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7652464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7652615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7652896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7653032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7653209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7653500Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7653798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7654090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7654258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7654424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7654589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7654764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7654937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7655113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7655391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7655690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7655980Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7656393Z STAGE:2023-01-11 21:17:03 24557:24557 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:17:03 24556:24556 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7656399Z 
2023-01-11T21:18:21.7656648Z STAGE:2023-01-11 21:17:03 24558:24558 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7656897Z STAGE:2023-01-11 21:17:03 24556:24556 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7657322Z STAGE:2023-01-11 21:17:03 24558:24558 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:17:03 24556:24556 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7657331Z 
2023-01-11T21:18:21.7657578Z STAGE:2023-01-11 21:17:03 24557:24557 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7657836Z STAGE:2023-01-11 21:17:03 24557:24557 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7658094Z STAGE:2023-01-11 21:17:03 24558:24558 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7658336Z STAGE:2023-01-11 21:17:03 24556:24556 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7658722Z STAGE:2023-01-11 21:17:03 24558:24558 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:17:03 24557:24557 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7658743Z 
2023-01-11T21:18:21.7658978Z STAGE:2023-01-11 21:17:03 24556:24556 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7659235Z STAGE:2023-01-11 21:17:03 24556:24556 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7659705Z STAGE:2023-01-11 21:17:03 24558:24558 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:17:03 24557:24557 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7659712Z 
2023-01-11T21:18:21.7660144Z STAGE:2023-01-11 21:17:03 24557:24557 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:17:03 24558:24558 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7660150Z 
2023-01-11T21:18:21.7660394Z STAGE:2023-01-11 21:17:03 24556:24556 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7660791Z STAGE:2023-01-11 21:17:03 24558:24558 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:17:03 24557:24557 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7660796Z 
2023-01-11T21:18:21.7661043Z STAGE:2023-01-11 21:17:03 24556:24556 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7661290Z STAGE:2023-01-11 21:17:03 24558:24558 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7661537Z STAGE:2023-01-11 21:17:03 24557:24557 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7661921Z STAGE:2023-01-11 21:17:03 24556:24556 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7662180Z STAGE:2023-01-11 21:17:03 24558:24558 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7662419Z STAGE:2023-01-11 21:17:03 24557:24557 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7662489Z ok (2.284s)
2023-01-11T21:18:21.7662494Z 
2023-01-11T21:18:21.7662696Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7662773Z Ran 1 test in 2.284s
2023-01-11T21:18:21.7662778Z 
2023-01-11T21:18:21.7662842Z OK
2023-01-11T21:18:21.7662847Z 
2023-01-11T21:18:21.7662932Z Generating XML reports...
2023-01-11T21:18:21.7663279Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211701.xml
2023-01-11T21:18:21.7663513Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7663779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7663907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7664191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7664327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7664332Z 
2023-01-11T21:18:21.7664409Z Running tests...
2023-01-11T21:18:21.7664608Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7664839Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24621
2023-01-11T21:18:21.7664999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24622
2023-01-11T21:18:21.7665155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24623
2023-01-11T21:18:21.7665420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7665546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7665831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7665968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7666144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7666492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7666800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7666926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7667210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7667332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7667505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7667780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7667903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7668186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7668322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7668497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7668798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7669082Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7669254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7669420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7669583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7669761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:18:21.7669935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:18:21.7670106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:18:21.7670402Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7670699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7670936Z STAGE:2023-01-11 21:17:07 24623:24623 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7671184Z STAGE:2023-01-11 21:17:07 24622:24622 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7671432Z STAGE:2023-01-11 21:17:07 24622:24622 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7671680Z STAGE:2023-01-11 21:17:07 24623:24623 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7671939Z STAGE:2023-01-11 21:17:07 24622:24622 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7672195Z STAGE:2023-01-11 21:17:07 24623:24623 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7672440Z STAGE:2023-01-11 21:17:07 24622:24622 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7672680Z STAGE:2023-01-11 21:17:07 24623:24623 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7672927Z STAGE:2023-01-11 21:17:07 24623:24623 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7673156Z STAGE:2023-01-11 21:17:07 24622:24622 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7673414Z STAGE:2023-01-11 21:17:07 24623:24623 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7673735Z STAGE:2023-01-11 21:17:07 24622:24622 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7674038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2023-01-11T21:18:21.7674108Z ok (2.275s)
2023-01-11T21:18:21.7674114Z 
2023-01-11T21:18:21.7674312Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7674388Z Ran 1 test in 2.275s
2023-01-11T21:18:21.7674394Z 
2023-01-11T21:18:21.7674454Z OK
2023-01-11T21:18:21.7674459Z 
2023-01-11T21:18:21.7674544Z Generating XML reports...
2023-01-11T21:18:21.7674876Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211705.xml
2023-01-11T21:18:21.7675114Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7675394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7675525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7675811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7675948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7675953Z 
2023-01-11T21:18:21.7676028Z Running tests...
2023-01-11T21:18:21.7676226Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7676462Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24682
2023-01-11T21:18:21.7676608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24683
2023-01-11T21:18:21.7676767Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24684
2023-01-11T21:18:21.7677045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7677173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7677459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7677594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7677773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7678045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7678155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7678439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7678579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7678757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7679030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7679154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7679437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7679572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7679748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7680034Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7680374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7680699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7680869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7681037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7681202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7681271Z ok (2.181s)
2023-01-11T21:18:21.7681276Z 
2023-01-11T21:18:21.7681475Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7681538Z Ran 1 test in 2.181s
2023-01-11T21:18:21.7681557Z 
2023-01-11T21:18:21.7681666Z OK
2023-01-11T21:18:21.7681672Z 
2023-01-11T21:18:21.7681760Z Generating XML reports...
2023-01-11T21:18:21.7682110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211709.xml
2023-01-11T21:18:21.7682348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7682627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7682754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7683036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7683174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7683179Z 
2023-01-11T21:18:21.7683239Z Running tests...
2023-01-11T21:18:21.7683438Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7683664Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24735
2023-01-11T21:18:21.7683824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24736
2023-01-11T21:18:21.7683986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24737
2023-01-11T21:18:21.7684261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7684386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7684667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7684804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7684970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7685276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7685550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7685678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7685961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7686096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7686273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7686547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7686658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7686939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7687111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7687314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7687615Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7687912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7688079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7688244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7688408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7688465Z ok (2.182s)
2023-01-11T21:18:21.7688470Z 
2023-01-11T21:18:21.7688672Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7688749Z Ran 1 test in 2.182s
2023-01-11T21:18:21.7688754Z 
2023-01-11T21:18:21.7688815Z OK
2023-01-11T21:18:21.7688821Z 
2023-01-11T21:18:21.7688910Z Generating XML reports...
2023-01-11T21:18:21.7689255Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211713.xml
2023-01-11T21:18:21.7689492Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7689771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7689901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7690170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7690309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7690317Z 
2023-01-11T21:18:21.7690391Z Running tests...
2023-01-11T21:18:21.7690590Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7690830Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24788
2023-01-11T21:18:21.7690988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24789
2023-01-11T21:18:21.7691144Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24790
2023-01-11T21:18:21.7691417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7691528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7691815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7691952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7692132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7692409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7692532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7692813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7692949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7693126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7693383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7693508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7693818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7693981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7694159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7694461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7694761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7695055Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7695222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7695370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7695537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7695606Z ok (2.170s)
2023-01-11T21:18:21.7695614Z 
2023-01-11T21:18:21.7695813Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7695890Z Ran 1 test in 2.170s
2023-01-11T21:18:21.7695896Z 
2023-01-11T21:18:21.7695957Z OK
2023-01-11T21:18:21.7695962Z 
2023-01-11T21:18:21.7696049Z Generating XML reports...
2023-01-11T21:18:21.7696397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211717.xml
2023-01-11T21:18:21.7696616Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7696891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7697014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7697300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7697438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7697443Z 
2023-01-11T21:18:21.7697518Z Running tests...
2023-01-11T21:18:21.7697718Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7697972Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24841
2023-01-11T21:18:21.7698130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24842
2023-01-11T21:18:21.7698273Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24843
2023-01-11T21:18:21.7698550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7698674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7698958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7699094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7699273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7699547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7699672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7699958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7700080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7700257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7700601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7700903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7701031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7701312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7701448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7701622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7702027Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7702327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7702495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7702664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7702829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7703080Z STAGE:2023-01-11 21:17:23 24841:24841 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7703323Z STAGE:2023-01-11 21:17:23 24843:24843 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7703562Z STAGE:2023-01-11 21:17:23 24842:24842 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7703812Z STAGE:2023-01-11 21:17:23 24841:24841 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7704057Z STAGE:2023-01-11 21:17:23 24841:24841 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7704309Z STAGE:2023-01-11 21:17:23 24842:24842 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7704559Z STAGE:2023-01-11 21:17:23 24843:24843 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7704818Z STAGE:2023-01-11 21:17:23 24842:24842 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7705073Z STAGE:2023-01-11 21:17:23 24843:24843 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7705141Z ok (2.177s)
2023-01-11T21:18:21.7705147Z 
2023-01-11T21:18:21.7705347Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7705426Z Ran 1 test in 2.177s
2023-01-11T21:18:21.7705432Z 
2023-01-11T21:18:21.7705493Z OK
2023-01-11T21:18:21.7705498Z 
2023-01-11T21:18:21.7705570Z Generating XML reports...
2023-01-11T21:18:21.7705914Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211721.xml
2023-01-11T21:18:21.7706153Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7706433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7706558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7706840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7706977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7706983Z 
2023-01-11T21:18:21.7707057Z Running tests...
2023-01-11T21:18:21.7707243Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7707497Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24897
2023-01-11T21:18:21.7707717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24898
2023-01-11T21:18:21.7707911Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24899
2023-01-11T21:18:21.7708191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7708315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7708597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7708732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7708911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7709171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7709298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7709579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7709714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7709892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7710164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7710287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7710569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7710703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7710866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7711172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7711472Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7711766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7711933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7712098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7712260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7712509Z STAGE:2023-01-11 21:17:27 24899:24899 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7712751Z STAGE:2023-01-11 21:17:27 24897:24897 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7712984Z STAGE:2023-01-11 21:17:27 24898:24898 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7713235Z STAGE:2023-01-11 21:17:27 24897:24897 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7713494Z STAGE:2023-01-11 21:17:27 24897:24897 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7713740Z STAGE:2023-01-11 21:17:27 24898:24898 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7713983Z STAGE:2023-01-11 21:17:27 24899:24899 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7714239Z STAGE:2023-01-11 21:17:27 24898:24898 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7714501Z STAGE:2023-01-11 21:17:27 24899:24899 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7714604Z ok (2.274s)
2023-01-11T21:18:21.7714610Z 
2023-01-11T21:18:21.7714808Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7714898Z Ran 1 test in 2.275s
2023-01-11T21:18:21.7714904Z 
2023-01-11T21:18:21.7714966Z OK
2023-01-11T21:18:21.7714971Z 
2023-01-11T21:18:21.7715055Z Generating XML reports...
2023-01-11T21:18:21.7715398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211725.xml
2023-01-11T21:18:21.7715631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7715906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7716034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7716319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7716443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7716463Z 
2023-01-11T21:18:21.7716525Z Running tests...
2023-01-11T21:18:21.7716724Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7716971Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24953
2023-01-11T21:18:21.7717127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24954
2023-01-11T21:18:21.7717283Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24955
2023-01-11T21:18:21.7717556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7717679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7718008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7718134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7718315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7718616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7718892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7719013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7719293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7719426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7719603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7719863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7719991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7720275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7720410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7720586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7720886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7721182Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7721374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7721539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7721789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7722045Z STAGE:2023-01-11 21:17:31 24953:24953 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7722289Z STAGE:2023-01-11 21:17:31 24954:24954 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7722525Z STAGE:2023-01-11 21:17:31 24955:24955 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7722775Z STAGE:2023-01-11 21:17:31 24955:24955 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7723034Z STAGE:2023-01-11 21:17:31 24955:24955 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7723283Z STAGE:2023-01-11 21:17:31 24953:24953 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7723529Z STAGE:2023-01-11 21:17:31 24954:24954 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7723787Z STAGE:2023-01-11 21:17:31 24953:24953 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7724033Z STAGE:2023-01-11 21:17:31 24954:24954 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7724101Z ok (2.173s)
2023-01-11T21:18:21.7724107Z 
2023-01-11T21:18:21.7724308Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7724385Z Ran 1 test in 2.173s
2023-01-11T21:18:21.7724391Z 
2023-01-11T21:18:21.7724453Z OK
2023-01-11T21:18:21.7724458Z 
2023-01-11T21:18:21.7724547Z Generating XML reports...
2023-01-11T21:18:21.7724895Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211730.xml
2023-01-11T21:18:21.7725130Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7725408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7725521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7725806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7725942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7725947Z 
2023-01-11T21:18:21.7726021Z Running tests...
2023-01-11T21:18:21.7726221Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7726393Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2023-01-11T21:18:21.7726399Z 
2023-01-11T21:18:21.7726598Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7726676Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7726681Z 
2023-01-11T21:18:21.7726741Z OK (skipped=1)
2023-01-11T21:18:21.7726759Z 
2023-01-11T21:18:21.7726834Z Generating XML reports...
2023-01-11T21:18:21.7727176Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211734.xml
2023-01-11T21:18:21.7727410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7727686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7727811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7728096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7728233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7728239Z 
2023-01-11T21:18:21.7728348Z Running tests...
2023-01-11T21:18:21.7728532Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7728756Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2023-01-11T21:18:21.7728763Z 
2023-01-11T21:18:21.7728958Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7729036Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7729042Z 
2023-01-11T21:18:21.7729115Z OK (skipped=1)
2023-01-11T21:18:21.7729120Z 
2023-01-11T21:18:21.7729206Z Generating XML reports...
2023-01-11T21:18:21.7729547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211735.xml
2023-01-11T21:18:21.7729782Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7730061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7730176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7730460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7730598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7730602Z 
2023-01-11T21:18:21.7730673Z Running tests...
2023-01-11T21:18:21.7730872Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7731060Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2023-01-11T21:18:21.7731066Z 
2023-01-11T21:18:21.7731261Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7731337Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7731343Z 
2023-01-11T21:18:21.7731401Z OK (skipped=1)
2023-01-11T21:18:21.7731422Z 
2023-01-11T21:18:21.7731494Z Generating XML reports...
2023-01-11T21:18:21.7731838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211737.xml
2023-01-11T21:18:21.7732073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7732347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7732472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7732757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7732898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7732903Z 
2023-01-11T21:18:21.7732977Z Running tests...
2023-01-11T21:18:21.7733161Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7733402Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25039
2023-01-11T21:18:21.7733567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25040
2023-01-11T21:18:21.7733726Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25041
2023-01-11T21:18:21.7734002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7734127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7734409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7734545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7734726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7734985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7735176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7735485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7735623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7735804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7736078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7736203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7736485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7736607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7736785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7737090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7737391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7737684Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7737851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7738017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7738181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7738433Z STAGE:2023-01-11 21:17:40 25039:25039 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7738665Z STAGE:2023-01-11 21:17:40 25040:25040 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7738912Z STAGE:2023-01-11 21:17:40 25041:25041 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7739162Z STAGE:2023-01-11 21:17:40 25041:25041 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7739423Z STAGE:2023-01-11 21:17:40 25041:25041 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7739670Z STAGE:2023-01-11 21:17:40 25039:25039 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7739913Z STAGE:2023-01-11 21:17:40 25040:25040 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7740172Z STAGE:2023-01-11 21:17:40 25039:25039 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7740430Z STAGE:2023-01-11 21:17:40 25040:25040 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7740503Z ok (2.175s)
2023-01-11T21:18:21.7740509Z 
2023-01-11T21:18:21.7740695Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7740776Z Ran 1 test in 2.176s
2023-01-11T21:18:21.7740782Z 
2023-01-11T21:18:21.7740845Z OK
2023-01-11T21:18:21.7740850Z 
2023-01-11T21:18:21.7740938Z Generating XML reports...
2023-01-11T21:18:21.7741313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211739.xml
2023-01-11T21:18:21.7741603Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7741992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7742119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7742404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7742591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7742597Z 
2023-01-11T21:18:21.7742704Z Running tests...
2023-01-11T21:18:21.7742908Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7743142Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25095
2023-01-11T21:18:21.7743303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25096
2023-01-11T21:18:21.7743461Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25097
2023-01-11T21:18:21.7743735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7743861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7744130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7744272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7744455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7744727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7744854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7745137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7745273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7745451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7745754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7746017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7746143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7746423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7746559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7746735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7747039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7747337Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7747504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7747675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7747830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7747903Z ok (2.178s)
2023-01-11T21:18:21.7747908Z 
2023-01-11T21:18:21.7748107Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7748184Z Ran 1 test in 2.178s
2023-01-11T21:18:21.7748190Z 
2023-01-11T21:18:21.7748251Z OK
2023-01-11T21:18:21.7748257Z 
2023-01-11T21:18:21.7748343Z Generating XML reports...
2023-01-11T21:18:21.7748689Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211743.xml
2023-01-11T21:18:21.7748923Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7749186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7749367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7749679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7749818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7749824Z 
2023-01-11T21:18:21.7749899Z Running tests...
2023-01-11T21:18:21.7750096Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7750348Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25148
2023-01-11T21:18:21.7750507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25149
2023-01-11T21:18:21.7750662Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25150
2023-01-11T21:18:21.7750922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7751050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7751333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7751470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7751648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7751923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7752049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7752330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7752454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7752633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7752936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7753206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7753331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7753613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7753747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7753923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7754222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7754510Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7754678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7754845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7755007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7755261Z STAGE:2023-01-11 21:17:48 25148:25148 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7755504Z STAGE:2023-01-11 21:17:48 25149:25149 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7755751Z STAGE:2023-01-11 21:17:48 25150:25150 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7756031Z STAGE:2023-01-11 21:17:48 25148:25148 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7756317Z STAGE:2023-01-11 21:17:48 25148:25148 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7756554Z STAGE:2023-01-11 21:17:48 25149:25149 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7756800Z STAGE:2023-01-11 21:17:48 25150:25150 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7757059Z STAGE:2023-01-11 21:17:48 25149:25149 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7757317Z STAGE:2023-01-11 21:17:48 25150:25150 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7757388Z ok (2.168s)
2023-01-11T21:18:21.7757395Z 
2023-01-11T21:18:21.7757594Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7757672Z Ran 1 test in 2.168s
2023-01-11T21:18:21.7757680Z 
2023-01-11T21:18:21.7757744Z OK
2023-01-11T21:18:21.7757749Z 
2023-01-11T21:18:21.7757837Z Generating XML reports...
2023-01-11T21:18:21.7758169Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211747.xml
2023-01-11T21:18:21.7758404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7758682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7758812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7759095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7759231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7759238Z 
2023-01-11T21:18:21.7759313Z Running tests...
2023-01-11T21:18:21.7759512Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7759752Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25204
2023-01-11T21:18:21.7759913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25205
2023-01-11T21:18:21.7760074Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25206
2023-01-11T21:18:21.7760349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7760475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7760761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7760896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7761078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7761354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7761467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7761819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7761957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7762138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7762442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7762718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7762842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7763158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7763325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7763488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7763791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7764091Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7764256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7764423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7764585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7764839Z STAGE:2023-01-11 21:17:52 25204:25204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7765085Z STAGE:2023-01-11 21:17:52 25205:25205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7765325Z STAGE:2023-01-11 21:17:52 25206:25206 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:18:21.7765558Z STAGE:2023-01-11 21:17:52 25204:25204 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7765816Z STAGE:2023-01-11 21:17:52 25204:25204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7766063Z STAGE:2023-01-11 21:17:52 25205:25205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7766309Z STAGE:2023-01-11 21:17:52 25206:25206 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:18:21.7766567Z STAGE:2023-01-11 21:17:52 25205:25205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7766826Z STAGE:2023-01-11 21:17:52 25206:25206 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:18:21.7766897Z ok (2.172s)
2023-01-11T21:18:21.7766904Z 
2023-01-11T21:18:21.7767105Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7767169Z Ran 1 test in 2.172s
2023-01-11T21:18:21.7767189Z 
2023-01-11T21:18:21.7767238Z OK
2023-01-11T21:18:21.7767242Z 
2023-01-11T21:18:21.7767328Z Generating XML reports...
2023-01-11T21:18:21.7767672Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211751.xml
2023-01-11T21:18:21.7767907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7768184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7768312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7768596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7768734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7768740Z 
2023-01-11T21:18:21.7768801Z Running tests...
2023-01-11T21:18:21.7769001Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7769237Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25260
2023-01-11T21:18:21.7769396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25261
2023-01-11T21:18:21.7769556Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25262
2023-01-11T21:18:21.7769830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7769990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7770301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7770439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7770604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7770879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7771004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7771286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7771426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7771604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7771882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7772008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7772274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7772415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7772588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7772887Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7773188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7773484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7773654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7773819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7773985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7774040Z ok (2.274s)
2023-01-11T21:18:21.7774047Z 
2023-01-11T21:18:21.7774248Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7774323Z Ran 1 test in 2.274s
2023-01-11T21:18:21.7774329Z 
2023-01-11T21:18:21.7774391Z OK
2023-01-11T21:18:21.7774396Z 
2023-01-11T21:18:21.7774482Z Generating XML reports...
2023-01-11T21:18:21.7774826Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211755.xml
2023-01-11T21:18:21.7775063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7775341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7775468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7775739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7775875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7775880Z 
2023-01-11T21:18:21.7775957Z Running tests...
2023-01-11T21:18:21.7776156Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7776402Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25340
2023-01-11T21:18:21.7776560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25341
2023-01-11T21:18:21.7776749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25342
2023-01-11T21:18:21.7777053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7777164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7777450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7777585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7777763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7778033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7778158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7778441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7778577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7778760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7779016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7779143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7779424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7779565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7779740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7780039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7780340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7780631Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7780796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7780943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7781106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7781202Z skip: CUDA is not available. (2.173s)
2023-01-11T21:18:21.7781208Z 
2023-01-11T21:18:21.7781407Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7781484Z Ran 1 test in 2.174s
2023-01-11T21:18:21.7781491Z 
2023-01-11T21:18:21.7781566Z OK (skipped=1)
2023-01-11T21:18:21.7781571Z 
2023-01-11T21:18:21.7781770Z Generating XML reports...
2023-01-11T21:18:21.7782124Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211759.xml
2023-01-11T21:18:21.7782346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7782623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7782747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7783028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7783165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7783170Z 
2023-01-11T21:18:21.7783246Z Running tests...
2023-01-11T21:18:21.7783447Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7783741Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25393
2023-01-11T21:18:21.7783932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25394
2023-01-11T21:18:21.7784076Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25395
2023-01-11T21:18:21.7784352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7784475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7784760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7784896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7785080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7785356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7785483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7785766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7785890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7786066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7786367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7786644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7786769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7787052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7787190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7787367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7787655Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7787956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7788121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7788281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7788446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7788553Z skip: Need at least 2 CUDA devices (2.183s)
2023-01-11T21:18:21.7788558Z 
2023-01-11T21:18:21.7788757Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7788834Z Ran 1 test in 2.184s
2023-01-11T21:18:21.7788839Z 
2023-01-11T21:18:21.7788911Z OK (skipped=1)
2023-01-11T21:18:21.7788916Z 
2023-01-11T21:18:21.7788989Z Generating XML reports...
2023-01-11T21:18:21.7789335Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211803.xml
2023-01-11T21:18:21.7789568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7789841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7789965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7790249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7790414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7790419Z 
2023-01-11T21:18:21.7790520Z Running tests...
2023-01-11T21:18:21.7790720Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7790942Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25446
2023-01-11T21:18:21.7791103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25447
2023-01-11T21:18:21.7791260Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25448
2023-01-11T21:18:21.7791532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7791656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7791939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7792075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7792254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7792516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7792686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7793043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7793181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7793362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7793635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7793760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7794042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7794179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7794343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7794645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7794944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7795238Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7795406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7795572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7795734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7795805Z ok (2.176s)
2023-01-11T21:18:21.7795810Z 
2023-01-11T21:18:21.7796010Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7796074Z Ran 1 test in 2.177s
2023-01-11T21:18:21.7796079Z 
2023-01-11T21:18:21.7796144Z OK
2023-01-11T21:18:21.7796149Z 
2023-01-11T21:18:21.7796235Z Generating XML reports...
2023-01-11T21:18:21.7796580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211807.xml
2023-01-11T21:18:21.7796815Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7797156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7797282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7797601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7797725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7797745Z 
2023-01-11T21:18:21.7797805Z Running tests...
2023-01-11T21:18:21.7798005Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7798236Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25499
2023-01-11T21:18:21.7798395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25500
2023-01-11T21:18:21.7798553Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25501
2023-01-11T21:18:21.7798829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7798956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7799237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7799358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7799539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7799815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7799938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7800221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7800360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7800538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7800842Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7801117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7801227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7801508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7801716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7801898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7802196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7802500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7802666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7802833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7802980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7803087Z skip: Need at least 2 CUDA devices (2.180s)
2023-01-11T21:18:21.7803092Z 
2023-01-11T21:18:21.7803294Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7803371Z Ran 1 test in 2.180s
2023-01-11T21:18:21.7803376Z 
2023-01-11T21:18:21.7803450Z OK (skipped=1)
2023-01-11T21:18:21.7803455Z 
2023-01-11T21:18:21.7803541Z Generating XML reports...
2023-01-11T21:18:21.7803922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211811.xml
2023-01-11T21:18:21.7804185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7804465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7804576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7804858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7804990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7804996Z 
2023-01-11T21:18:21.7805072Z Running tests...
2023-01-11T21:18:21.7805270Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7805527Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25552
2023-01-11T21:18:21.7805692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25553
2023-01-11T21:18:21.7805850Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25554
2023-01-11T21:18:21.7806123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7806234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7806521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7806657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7806837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:18:21.7807109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7807236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7807523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7807659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7807823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:18:21.7808098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7808223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7808505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7808640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7808819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:18:21.7809120Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7809420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7809718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2023-01-11T21:18:21.7809870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:18:21.7810036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:18:21.7810200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:18:21.7810305Z skip: Need at least 2 CUDA devices (2.280s)
2023-01-11T21:18:21.7810353Z 
2023-01-11T21:18:21.7810554Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7810631Z Ran 1 test in 2.281s
2023-01-11T21:18:21.7810637Z 
2023-01-11T21:18:21.7810739Z OK (skipped=1)
2023-01-11T21:18:21.7810745Z 
2023-01-11T21:18:21.7810831Z Generating XML reports...
2023-01-11T21:18:21.7811176Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211815.xml
2023-01-11T21:18:21.7811396Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7811674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7811798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7812083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7812223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7812228Z 
2023-01-11T21:18:21.7812302Z Running tests...
2023-01-11T21:18:21.7812505Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7812853Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s)
2023-01-11T21:18:21.7812860Z 
2023-01-11T21:18:21.7813058Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7813123Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7813128Z 
2023-01-11T21:18:21.7813203Z OK (skipped=1)
2023-01-11T21:18:21.7813208Z 
2023-01-11T21:18:21.7813294Z Generating XML reports...
2023-01-11T21:18:21.7813634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211819.xml
2023-01-11T21:18:21.7813866Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:18:21.7814142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:18:21.7814270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:18:21.7814556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:18:21.7814679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:18:21.7814698Z 
2023-01-11T21:18:21.7814760Z Running tests...
2023-01-11T21:18:21.7814957Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7815304Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s)
2023-01-11T21:18:21.7815310Z 
2023-01-11T21:18:21.7815502Z ----------------------------------------------------------------------
2023-01-11T21:18:21.7815582Z Ran 1 test in 0.002s
2023-01-11T21:18:21.7815587Z 
2023-01-11T21:18:21.7815659Z OK (skipped=1)
2023-01-11T21:18:21.7815664Z 
2023-01-11T21:18:21.7815751Z Generating XML reports...
2023-01-11T21:18:21.7816093Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211820.xml
2023-01-11T21:18:21.7816098Z 
2023-01-11T21:18:21.7816479Z ##[endgroup]
2023-01-11T21:18:21.7816837Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_5uyus5i3)
2023-01-11T21:18:21.7816843Z 
2023-01-11T21:18:21.7817024Z Running distributed/test_nccl ... [2023-01-11 21:18:21.524469]
2023-01-11T21:18:21.7817368Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:18:21.524765]
2023-01-11T21:18:23.3527226Z 
2023-01-11T21:18:23.3527703Z Expand the folded group to see the log file of distributed/test_nccl
2023-01-11T21:18:23.3528864Z ##[group]PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_vyxormwp)
2023-01-11T21:18:23.3529284Z CUDA not available, skipping tests
2023-01-11T21:18:23.3529420Z 
2023-01-11T21:18:23.3529494Z Running tests...
2023-01-11T21:18:23.3529892Z ----------------------------------------------------------------------
2023-01-11T21:18:23.3530067Z 
2023-01-11T21:18:23.3530264Z ----------------------------------------------------------------------
2023-01-11T21:18:23.3530505Z Ran 0 tests in 0.000s
2023-01-11T21:18:23.3530624Z 
2023-01-11T21:18:23.3530695Z OK
2023-01-11T21:18:23.3530830Z 
2023-01-11T21:18:23.3530965Z Generating XML reports...
2023-01-11T21:18:23.3531374Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl
2023-01-11T21:18:23.3531567Z 
2023-01-11T21:18:23.3531796Z ##[endgroup]
2023-01-11T21:18:23.3532194Z FINISHED PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_vyxormwp)
2023-01-11T21:18:23.3532429Z 
2023-01-11T21:18:23.3532673Z Running distributed/pipeline/sync/test_transparency ... [2023-01-11 21:18:23.352954]
2023-01-11T21:18:23.3534526Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2023-01-11 21:18:23.353294]
2023-01-11T21:18:25.6856381Z 
2023-01-11T21:18:25.6856961Z Expand the folded group to see the log file of distributed/pipeline/sync/test_transparency
2023-01-11T21:18:25.6858131Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency_d3u0un6x)
2023-01-11T21:18:25.6858577Z ============================= test session starts ==============================
2023-01-11T21:18:25.6859061Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T21:18:25.6859323Z cachedir: .pytest_cache
2023-01-11T21:18:25.6860003Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T21:18:25.6860468Z torch: 2.0.0a0+git8419ddd
2023-01-11T21:18:25.6860713Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T21:18:25.6861135Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T21:18:25.6861437Z collecting ... collected 1 item
2023-01-11T21:18:25.6861932Z Running 1 items in this shard: test/distributed/pipeline/sync/test_transparency.py::test_simple_linears
2023-01-11T21:18:25.6862139Z 
2023-01-11T21:18:25.6862304Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%]
2023-01-11T21:18:25.6862479Z 
2023-01-11T21:18:25.6862591Z ============================== 1 passed in 0.08s ===============================
2023-01-11T21:18:25.6862731Z 
2023-01-11T21:18:25.6862975Z ##[endgroup]
2023-01-11T21:18:25.6863631Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency_d3u0un6x)
2023-01-11T21:18:25.6863997Z 
2023-01-11T21:18:25.6864339Z Running distributed/pipeline/sync/test_pipeline ... [2023-01-11 21:18:25.685897]
2023-01-11T21:18:25.6864817Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2023-01-11 21:18:25.686276]
2023-01-11T21:18:27.4050887Z 
2023-01-11T21:18:27.4051468Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipeline
2023-01-11T21:18:27.4052759Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipeline_szi54glz)
2023-01-11T21:18:27.4053220Z ============================= test session starts ==============================
2023-01-11T21:18:27.4054041Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T21:18:27.4054517Z cachedir: .pytest_cache
2023-01-11T21:18:27.4055012Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T21:18:27.4055325Z torch: 2.0.0a0+git8419ddd
2023-01-11T21:18:27.4055562Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T21:18:27.4055994Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T21:18:27.4056297Z collecting ... collected 1 item
2023-01-11T21:18:27.4056843Z Running 1 items in this shard: test/distributed/pipeline/sync/test_pipeline.py::test_clock_cycles
2023-01-11T21:18:27.4057134Z 
2023-01-11T21:18:27.4057288Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED     [100%]
2023-01-11T21:18:27.4057472Z 
2023-01-11T21:18:27.4057580Z ============================== 1 passed in 0.02s ===============================
2023-01-11T21:18:27.4057715Z 
2023-01-11T21:18:27.4057939Z ##[endgroup]
2023-01-11T21:18:27.4058430Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipeline_szi54glz)
2023-01-11T21:18:27.4058716Z 
2023-01-11T21:18:27.4058926Z Running distributed/pipeline/sync/test_phony ... [2023-01-11 21:18:27.405478]
2023-01-11T21:18:27.4059907Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2023-01-11 21:18:27.405844]
2023-01-11T21:18:29.1511223Z 
2023-01-11T21:18:29.1511822Z Expand the folded group to see the log file of distributed/pipeline/sync/test_phony
2023-01-11T21:18:29.1513029Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_xmo0jlmc)
2023-01-11T21:18:29.1513752Z ============================= test session starts ==============================
2023-01-11T21:18:29.1514498Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T21:18:29.1514964Z cachedir: .pytest_cache
2023-01-11T21:18:29.1515634Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T21:18:29.1516139Z torch: 2.0.0a0+git8419ddd
2023-01-11T21:18:29.1516528Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T21:18:29.1517252Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T21:18:29.1517695Z collecting ... collected 4 items
2023-01-11T21:18:29.1518553Z Running 4 items in this shard: test/distributed/pipeline/sync/test_phony.py::test_phony_size, test/distributed/pipeline/sync/test_phony.py::test_phony_requires_grad, test/distributed/pipeline/sync/test_phony.py::test_cached_phony, test/distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function
2023-01-11T21:18:29.1519185Z 
2023-01-11T21:18:29.1519445Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED          [ 25%]
2023-01-11T21:18:29.1519962Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%]
2023-01-11T21:18:29.1520495Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED        [ 75%]
2023-01-11T21:18:29.1521010Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%]
2023-01-11T21:18:29.1521299Z 
2023-01-11T21:18:29.1521455Z ============================== 4 passed in 0.03s ===============================
2023-01-11T21:18:29.1521733Z 
2023-01-11T21:18:29.1522100Z ##[endgroup]
2023-01-11T21:18:29.1522807Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_xmo0jlmc)
2023-01-11T21:18:29.1523207Z 
2023-01-11T21:18:29.1523586Z Running distributed/pipeline/sync/test_inplace ... [2023-01-11 21:18:29.151530]
2023-01-11T21:18:29.1524377Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2023-01-11 21:18:29.151941]
2023-01-11T21:18:30.9361434Z 
2023-01-11T21:18:30.9362304Z Expand the folded group to see the log file of distributed/pipeline/sync/test_inplace
2023-01-11T21:18:30.9363595Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_8rtz05gp)
2023-01-11T21:18:30.9364327Z ============================= test session starts ==============================
2023-01-11T21:18:30.9365144Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T21:18:30.9365443Z cachedir: .pytest_cache
2023-01-11T21:18:30.9365919Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T21:18:30.9366256Z torch: 2.0.0a0+git8419ddd
2023-01-11T21:18:30.9366522Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T21:18:30.9366950Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T21:18:30.9367244Z collecting ... collected 3 items
2023-01-11T21:18:30.9367722Z Running 3 items in this shard: test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad
2023-01-11T21:18:30.9368079Z 
2023-01-11T21:18:30.9368243Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%]
2023-01-11T21:18:30.9368614Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%]
2023-01-11T21:18:30.9369031Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%]
2023-01-11T21:18:30.9369239Z 
2023-01-11T21:18:30.9369392Z ========================= 1 passed, 2 xfailed in 0.11s =========================
2023-01-11T21:18:30.9369579Z 
2023-01-11T21:18:30.9369804Z ##[endgroup]
2023-01-11T21:18:30.9370297Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_8rtz05gp)
2023-01-11T21:18:30.9370588Z 
2023-01-11T21:18:30.9370816Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2023-01-11 21:18:30.936457]
2023-01-11T21:18:30.9371303Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2023-01-11 21:18:30.936797]
2023-01-11T21:18:33.2934745Z 
2023-01-11T21:18:33.2935384Z Expand the folded group to see the log file of distributed/pipeline/sync/test_deferred_batch_norm
2023-01-11T21:18:33.2936634Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_fr4w0nsx)
2023-01-11T21:18:33.2937431Z ============================= test session starts ==============================
2023-01-11T21:18:33.2938198Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T21:18:33.2938589Z cachedir: .pytest_cache
2023-01-11T21:18:33.2939349Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T21:18:33.2939931Z torch: 2.0.0a0+git8419ddd
2023-01-11T21:18:33.2940346Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T21:18:33.2940770Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T21:18:33.2941069Z collecting ... collected 11 items
2023-01-11T21:18:33.2943173Z Running 11 items in this shard: test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad
2023-01-11T21:18:33.2944253Z 
2023-01-11T21:18:33.2944499Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [  9%]
2023-01-11T21:18:33.2944945Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%]
2023-01-11T21:18:33.2945378Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%]
2023-01-11T21:18:33.2945819Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%]
2023-01-11T21:18:33.2946181Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%]
2023-01-11T21:18:33.2946532Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%]
2023-01-11T21:18:33.2946896Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%]
2023-01-11T21:18:33.2947243Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED  [ 72%]
2023-01-11T21:18:33.2947562Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%]
2023-01-11T21:18:33.2947892Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%]
2023-01-11T21:18:33.2948237Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%]
2023-01-11T21:18:33.2948434Z 
2023-01-11T21:18:33.2948535Z ============================== 11 passed in 0.66s ==============================
2023-01-11T21:18:33.2948673Z 
2023-01-11T21:18:33.2948930Z ##[endgroup]
2023-01-11T21:18:33.2949459Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_fr4w0nsx)
2023-01-11T21:18:33.2949765Z 
2023-01-11T21:18:33.2949983Z Running distributed/pipeline/sync/test_checkpoint ... [2023-01-11 21:18:33.293781]
2023-01-11T21:18:33.2950447Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2023-01-11 21:18:33.294125]
2023-01-11T21:18:35.0491622Z 
2023-01-11T21:18:35.0492288Z Expand the folded group to see the log file of distributed/pipeline/sync/test_checkpoint
2023-01-11T21:18:35.0493511Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_7yd28w8q)
2023-01-11T21:18:35.0494308Z ============================= test session starts ==============================
2023-01-11T21:18:35.0495029Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T21:18:35.0495287Z cachedir: .pytest_cache
2023-01-11T21:18:35.0495761Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T21:18:35.0496114Z torch: 2.0.0a0+git8419ddd
2023-01-11T21:18:35.0496414Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T21:18:35.0496894Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T21:18:35.0497191Z collecting ... collected 7 items
2023-01-11T21:18:35.0498249Z Running 7 items in this shard: test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad, test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter, test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing, test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint, test/distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output
2023-01-11T21:18:35.0499066Z 
2023-01-11T21:18:35.0499226Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 14%]
2023-01-11T21:18:35.0499658Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 28%]
2023-01-11T21:18:35.0500035Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 42%]
2023-01-11T21:18:35.0500393Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 57%]
2023-01-11T21:18:35.0500749Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 71%]
2023-01-11T21:18:35.0501150Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 85%]
2023-01-11T21:18:35.0501525Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%]
2023-01-11T21:18:35.0501935Z 
2023-01-11T21:18:35.0502050Z ============================== 7 passed in 0.03s ===============================
2023-01-11T21:18:35.0502174Z 
2023-01-11T21:18:35.0502432Z ##[endgroup]
2023-01-11T21:18:35.0502946Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_7yd28w8q)
2023-01-11T21:18:35.0503234Z 
2023-01-11T21:18:35.0503451Z Running distributed/pipeline/sync/test_balance ... [2023-01-11 21:18:35.049501]
2023-01-11T21:18:35.0503899Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2023-01-11 21:18:35.049858]
2023-01-11T21:18:40.7923259Z 
2023-01-11T21:18:40.7923853Z Expand the folded group to see the log file of distributed/pipeline/sync/test_balance
2023-01-11T21:18:40.7924868Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_uave2k6j)
2023-01-11T21:18:40.7925293Z ============================= test session starts ==============================
2023-01-11T21:18:40.7925799Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T21:18:40.7926056Z cachedir: .pytest_cache
2023-01-11T21:18:40.7926537Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T21:18:40.7927039Z torch: 2.0.0a0+git8419ddd
2023-01-11T21:18:40.7927456Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T21:18:40.7928151Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T21:18:40.7928592Z collecting ... collected 15 items
2023-01-11T21:18:40.7930717Z Running 15 items in this shard: test/distributed/pipeline/sync/test_balance.py::test_blockpartition, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale, test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu], test/distributed/pipeline/sync/test_balance.py::test_not_training, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple, test/distributed/pipeline/sync/test_balance.py::test_already_has_grad
2023-01-11T21:18:40.7932167Z 
2023-01-11T21:18:40.7932412Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED    [  6%]
2023-01-11T21:18:40.7932834Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 13%]
2023-01-11T21:18:40.7933234Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 20%]
2023-01-11T21:18:40.7933604Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 26%]
2023-01-11T21:18:40.7933950Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 33%]
2023-01-11T21:18:40.7934282Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 40%]
2023-01-11T21:18:40.7934632Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent SKIPPED [ 46%]
2023-01-11T21:18:40.7934971Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param SKIPPED [ 53%]
2023-01-11T21:18:40.7935316Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale SKIPPED [ 60%]
2023-01-11T21:18:40.7935640Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 66%]
2023-01-11T21:18:40.7935984Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 73%]
2023-01-11T21:18:40.7936320Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED      [ 80%]
2023-01-11T21:18:40.7936633Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 86%]
2023-01-11T21:18:40.7936964Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple SKIPPED [ 93%]
2023-01-11T21:18:40.7937288Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED  [100%]
2023-01-11T21:18:40.7937467Z 
2023-01-11T21:18:40.7937583Z ======================== 10 passed, 5 skipped in 4.06s =========================
2023-01-11T21:18:40.7937725Z 
2023-01-11T21:18:40.7937984Z ##[endgroup]
2023-01-11T21:18:40.7938527Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_uave2k6j)
2023-01-11T21:18:40.7938809Z 
2023-01-11T21:18:40.7939029Z Running distributed/pipeline/sync/skip/test_tracker ... [2023-01-11 21:18:40.792741]
2023-01-11T21:18:40.7939490Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2023-01-11 21:18:40.793176]
2023-01-11T21:18:42.5303927Z 
2023-01-11T21:18:42.5304600Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_tracker
2023-01-11T21:18:42.5305818Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_tracker_8f378h5w)
2023-01-11T21:18:42.5306578Z ============================= test session starts ==============================
2023-01-11T21:18:42.5307228Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T21:18:42.5307486Z cachedir: .pytest_cache
2023-01-11T21:18:42.5307914Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T21:18:42.5308223Z torch: 2.0.0a0+git8419ddd
2023-01-11T21:18:42.5308482Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T21:18:42.5308946Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T21:18:42.5309229Z collecting ... collected 6 items
2023-01-11T21:18:42.5310159Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker, test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel, test/distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing
2023-01-11T21:18:42.5310829Z 
2023-01-11T21:18:42.5310998Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%]
2023-01-11T21:18:42.5311366Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel SKIPPED [ 33%]
2023-01-11T21:18:42.5311722Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%]
2023-01-11T21:18:42.5312090Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%]
2023-01-11T21:18:42.5312452Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%]
2023-01-11T21:18:42.5312898Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%]
2023-01-11T21:18:42.5313099Z 
2023-01-11T21:18:42.5313222Z ========================= 5 passed, 1 skipped in 0.03s =========================
2023-01-11T21:18:42.5313363Z 
2023-01-11T21:18:42.5313586Z ##[endgroup]
2023-01-11T21:18:42.5314101Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_tracker_8f378h5w)
2023-01-11T21:18:42.5314404Z 
2023-01-11T21:18:42.5314628Z Running distributed/pipeline/sync/skip/test_portal ... [2023-01-11 21:18:42.530801]
2023-01-11T21:18:42.5315093Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2023-01-11 21:18:42.531157]
2023-01-11T21:18:44.2592932Z 
2023-01-11T21:18:44.2593539Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_portal
2023-01-11T21:18:44.2594612Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_29xhsd1t)
2023-01-11T21:18:44.2595037Z ============================= test session starts ==============================
2023-01-11T21:18:44.2595508Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T21:18:44.2595769Z cachedir: .pytest_cache
2023-01-11T21:18:44.2596299Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T21:18:44.2596630Z torch: 2.0.0a0+git8419ddd
2023-01-11T21:18:44.2596866Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T21:18:44.2597282Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T21:18:44.2597581Z collecting ... collected 10 items
2023-01-11T21:18:44.2598638Z Running 10 items in this shard: test/distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad, test/distributed/pipeline/sync/skip/test_portal.py::test_use_grad, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1
2023-01-11T21:18:44.2599625Z 
2023-01-11T21:18:44.2599786Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device SKIPPED [ 10%]
2023-01-11T21:18:44.2600324Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED   [ 20%]
2023-01-11T21:18:44.2600723Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%]
2023-01-11T21:18:44.2601058Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED      [ 40%]
2023-01-11T21:18:44.2601383Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%]
2023-01-11T21:18:44.2601812Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%]
2023-01-11T21:18:44.2602166Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%]
2023-01-11T21:18:44.2602502Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%]
2023-01-11T21:18:44.2602849Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%]
2023-01-11T21:18:44.2603208Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%]
2023-01-11T21:18:44.2603411Z 
2023-01-11T21:18:44.2603529Z ========================= 9 passed, 1 skipped in 0.04s =========================
2023-01-11T21:18:44.2603675Z 
2023-01-11T21:18:44.2603899Z ##[endgroup]
2023-01-11T21:18:44.2604409Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_29xhsd1t)
2023-01-11T21:18:44.2604705Z 
2023-01-11T21:18:44.2604958Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2023-01-11 21:18:44.259545]
2023-01-11T21:18:44.2605459Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2023-01-11 21:18:44.259873]
2023-01-11T21:18:45.9767713Z 
2023-01-11T21:18:45.9768257Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_inspect_skip_layout
2023-01-11T21:18:45.9769175Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_7w30f2ee)
2023-01-11T21:18:45.9769699Z ============================= test session starts ==============================
2023-01-11T21:18:45.9770221Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T21:18:45.9770467Z cachedir: .pytest_cache
2023-01-11T21:18:45.9770942Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T21:18:45.9771269Z torch: 2.0.0a0+git8419ddd
2023-01-11T21:18:45.9771496Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T21:18:45.9772017Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T21:18:45.9772450Z collecting ... collected 6 items
2023-01-11T21:18:45.9773239Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace
2023-01-11T21:18:45.9773879Z 
2023-01-11T21:18:45.9774052Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%]
2023-01-11T21:18:45.9774400Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%]
2023-01-11T21:18:45.9774797Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%]
2023-01-11T21:18:45.9775219Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%]
2023-01-11T21:18:45.9775830Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%]
2023-01-11T21:18:45.9776249Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%]
2023-01-11T21:18:45.9776441Z 
2023-01-11T21:18:45.9776552Z ============================== 6 passed in 0.03s ===============================
2023-01-11T21:18:45.9776687Z 
2023-01-11T21:18:45.9776922Z ##[endgroup]
2023-01-11T21:18:45.9777465Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_7w30f2ee)
2023-01-11T21:18:45.9777785Z 
2023-01-11T21:18:45.9778001Z Running distributed/pipeline/sync/skip/test_api ... [2023-01-11 21:18:45.977082]
2023-01-11T21:18:45.9778460Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2023-01-11 21:18:45.977406]
2023-01-11T21:18:47.6938823Z 
2023-01-11T21:18:47.6939414Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_api
2023-01-11T21:18:47.6940405Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_api_eo89u5ru)
2023-01-11T21:18:47.6940815Z ============================= test session starts ==============================
2023-01-11T21:18:47.6941314Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T21:18:47.6941561Z cachedir: .pytest_cache
2023-01-11T21:18:47.6942182Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T21:18:47.6942583Z torch: 2.0.0a0+git8419ddd
2023-01-11T21:18:47.6942857Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T21:18:47.6943299Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T21:18:47.6943635Z collecting ... collected 3 items
2023-01-11T21:18:47.6944118Z Running 3 items in this shard: test/distributed/pipeline/sync/skip/test_api.py::test_namespace_difference, test/distributed/pipeline/sync/skip/test_api.py::test_namespace_copy, test/distributed/pipeline/sync/skip/test_api.py::test_skippable_repr
2023-01-11T21:18:47.6944454Z 
2023-01-11T21:18:47.6944618Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%]
2023-01-11T21:18:47.6944940Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED   [ 66%]
2023-01-11T21:18:47.6945260Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED   [100%]
2023-01-11T21:18:47.6945438Z 
2023-01-11T21:18:47.6945548Z ============================== 3 passed in 0.02s ===============================
2023-01-11T21:18:47.6945687Z 
2023-01-11T21:18:47.6945905Z ##[endgroup]
2023-01-11T21:18:47.6946546Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_api_eo89u5ru)
2023-01-11T21:18:47.6946839Z 
2023-01-11T21:18:47.6947076Z Running distributed/optim/test_apply_optimizer_in_backward ... [2023-01-11 21:18:47.694190]
2023-01-11T21:18:47.6947698Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:18:47.694568]
2023-01-11T21:18:49.2002687Z 
2023-01-11T21:18:49.2003359Z Expand the folded group to see the log file of distributed/optim/test_apply_optimizer_in_backward
2023-01-11T21:18:49.2004456Z ##[group]PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_0vm9ep9o)
2023-01-11T21:18:49.2004770Z 
2023-01-11T21:18:49.2005043Z ##[endgroup]
2023-01-11T21:18:49.2005690Z FINISHED PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_0vm9ep9o)
2023-01-11T21:18:49.2006273Z 
2023-01-11T21:18:49.2007178Z Running distributed/elastic/timer/api_test ... [2023-01-11 21:18:49.200523]
2023-01-11T21:18:49.2010471Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:18:49.200857]
2023-01-11T21:18:50.5264831Z 
2023-01-11T21:18:50.5265500Z Expand the folded group to see the log file of distributed/elastic/timer/api_test
2023-01-11T21:18:50.5266567Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-api_test_mm2qapt5)
2023-01-11T21:18:50.5266842Z 
2023-01-11T21:18:50.5267062Z ##[endgroup]
2023-01-11T21:18:50.5267635Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-api_test_mm2qapt5)
2023-01-11T21:18:50.5267926Z 
2023-01-11T21:18:50.5269550Z Running distributed/_shard/test_replicated_tensor ... [2023-01-11 21:18:50.526775]
2023-01-11T21:18:50.5273159Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_replicated_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:18:50.527129]
2023-01-11T21:18:52.0313194Z 
2023-01-11T21:18:52.0313861Z Expand the folded group to see the log file of distributed/_shard/test_replicated_tensor
2023-01-11T21:18:52.0314867Z ##[group]PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_6ww1h_3s)
2023-01-11T21:18:52.0315157Z 
2023-01-11T21:18:52.0315361Z ##[endgroup]
2023-01-11T21:18:52.0315929Z FINISHED PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_6ww1h_3s)
2023-01-11T21:18:52.0316217Z 
2023-01-11T21:18:52.0317830Z Running distributed/checkpoint/test_nested_dict ... [2023-01-11 21:18:52.031604]
2023-01-11T21:18:52.0322272Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:18:52.031985]
2023-01-11T21:18:53.9068019Z 
2023-01-11T21:18:53.9068610Z Expand the folded group to see the log file of distributed/checkpoint/test_nested_dict
2023-01-11T21:18:53.9069967Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_nested_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_nested_dict_dmi7xh9j)
2023-01-11T21:18:53.9070642Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_nested_dict
2023-01-11T21:18:53.9070864Z 
2023-01-11T21:18:53.9070942Z Running tests...
2023-01-11T21:18:53.9071247Z ----------------------------------------------------------------------
2023-01-11T21:18:53.9071579Z   test_flattening_round_trip (__main__.TestFlattening) ... ok (0.263s)
2023-01-11T21:18:53.9072011Z   test_mapping (__main__.TestFlattening) ... ok (0.002s)
2023-01-11T21:18:53.9072174Z 
2023-01-11T21:18:53.9072371Z ----------------------------------------------------------------------
2023-01-11T21:18:53.9072618Z Ran 2 tests in 0.265s
2023-01-11T21:18:53.9072732Z 
2023-01-11T21:18:53.9072794Z OK
2023-01-11T21:18:53.9072885Z 
2023-01-11T21:18:53.9072968Z Generating XML reports...
2023-01-11T21:18:53.9073431Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_nested_dict/TEST-TestFlattening-20230111211853.xml
2023-01-11T21:18:53.9073697Z 
2023-01-11T21:18:53.9073931Z ##[endgroup]
2023-01-11T21:18:53.9074416Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_nested_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_nested_dict_dmi7xh9j)
2023-01-11T21:18:53.9074694Z 
2023-01-11T21:18:53.9075138Z Running distributed/checkpoint/test_dedup_tensors ... [2023-01-11 21:18:53.907081]
2023-01-11T21:18:53.9076194Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:18:53.907435]
2023-01-11T21:18:55.7727246Z 
2023-01-11T21:18:55.7727666Z Expand the folded group to see the log file of distributed/checkpoint/test_dedup_tensors
2023-01-11T21:18:55.7728957Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_dedup_tensors (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_dedup_tensors_dwpplxa6)
2023-01-11T21:18:55.7729761Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors
2023-01-11T21:18:55.7729967Z 
2023-01-11T21:18:55.7730043Z Running tests...
2023-01-11T21:18:55.7730355Z ----------------------------------------------------------------------
2023-01-11T21:18:55.7730977Z   test_dedup_shards (__main__.TestDedupTensor) ... 2023-01-11 21:18:55,429 _dedup_tensors.py:44 INFO p:MainProcess t:MainThread: Duplicate keys to remove: {1: [MetadataIndex(fqn='tensor_0', offset=torch.Size([0]), index=None)]}
2023-01-11T21:18:55.7731355Z ok (0.262s)
2023-01-11T21:18:55.7731445Z 
2023-01-11T21:18:55.7731649Z ----------------------------------------------------------------------
2023-01-11T21:18:55.7731907Z Ran 1 test in 0.262s
2023-01-11T21:18:55.7732036Z 
2023-01-11T21:18:55.7732105Z OK
2023-01-11T21:18:55.7732232Z 
2023-01-11T21:18:55.7732306Z Generating XML reports...
2023-01-11T21:18:55.7732773Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors/TEST-TestDedupTensor-20230111211855.xml
2023-01-11T21:18:55.7733093Z 
2023-01-11T21:18:55.7733318Z ##[endgroup]
2023-01-11T21:18:55.7733791Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_dedup_tensors (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_dedup_tensors_dwpplxa6)
2023-01-11T21:18:55.7734079Z 
2023-01-11T21:18:55.7734286Z Running distributed/checkpoint/test_traverse ... [2023-01-11 21:18:55.773133]
2023-01-11T21:18:55.7736967Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:18:55.773518]
2023-01-11T21:18:57.6214095Z 
2023-01-11T21:18:57.6214715Z Expand the folded group to see the log file of distributed/checkpoint/test_traverse
2023-01-11T21:18:57.6216028Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_traverse_sa3itx02)
2023-01-11T21:18:57.6217121Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_traverse
2023-01-11T21:18:57.6217495Z 
2023-01-11T21:18:57.6217625Z Running tests...
2023-01-11T21:18:57.6218121Z ----------------------------------------------------------------------
2023-01-11T21:18:57.6218524Z   test_get_element (__main__.TestTraverse) ... ok (0.258s)
2023-01-11T21:18:57.6218920Z   test_set_element (__main__.TestTraverse) ... ok (0.002s)
2023-01-11T21:18:57.6219430Z   test_traverse_doesnt_ignore_intermediate_collections (__main__.TestTraverse) ... ok (0.002s)
2023-01-11T21:18:57.6219959Z   test_traverse_nested_dict (__main__.TestTraverse) ... ok (0.001s)
2023-01-11T21:18:57.6220397Z   test_traverse_nested_list (__main__.TestTraverse) ... ok (0.002s)
2023-01-11T21:18:57.6220805Z   test_traverse_shallow (__main__.TestTraverse) ... ok (0.001s)
2023-01-11T21:18:57.6221267Z   test_traverse_with_ordered_dict (__main__.TestTraverse) ... ok (0.001s)
2023-01-11T21:18:57.6221535Z 
2023-01-11T21:18:57.6222142Z ----------------------------------------------------------------------
2023-01-11T21:18:57.6222504Z Ran 7 tests in 0.267s
2023-01-11T21:18:57.6222682Z 
2023-01-11T21:18:57.6222776Z OK
2023-01-11T21:18:57.6222914Z 
2023-01-11T21:18:57.6223047Z Generating XML reports...
2023-01-11T21:18:57.6223787Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_traverse/TEST-TestTraverse-20230111211857.xml
2023-01-11T21:18:57.6224482Z 
2023-01-11T21:18:57.6224877Z ##[endgroup]
2023-01-11T21:18:57.6225798Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_traverse_sa3itx02)
2023-01-11T21:18:57.6226175Z 
2023-01-11T21:18:57.6226448Z Running distributed/fsdp/test_utils ... [2023-01-11 21:18:57.621814]
2023-01-11T21:18:57.6227283Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:18:57.622277]
2023-01-11T21:18:59.5041303Z 
2023-01-11T21:18:59.5041913Z Expand the folded group to see the log file of distributed/fsdp/test_utils
2023-01-11T21:18:59.5042986Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_q64npkhd)
2023-01-11T21:18:59.5043596Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils
2023-01-11T21:18:59.5043819Z 
2023-01-11T21:18:59.5043895Z Running tests...
2023-01-11T21:18:59.5044215Z ----------------------------------------------------------------------
2023-01-11T21:18:59.5044536Z   test_get_fully_sharded_module_to_states (__main__.TestGetSubmoduleToStates)
2023-01-11T21:18:59.5044866Z Tests the helper function ``_get_fully_sharded_module_states()`` that ... ok (0.258s)
2023-01-11T21:18:59.5045264Z   test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... skip: Skipped due to lack of GPU (0.002s)
2023-01-11T21:18:59.5045736Z   test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.003s)
2023-01-11T21:18:59.5046151Z   test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... skip: Skipped due to lack of GPU (0.002s)
2023-01-11T21:18:59.5046478Z   test_packed_sequence (__main__.TestUtils)
2023-01-11T21:18:59.5046847Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.002s)
2023-01-11T21:18:59.5047121Z   test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s)
2023-01-11T21:18:59.5047278Z 
2023-01-11T21:18:59.5047488Z ----------------------------------------------------------------------
2023-01-11T21:18:59.5047746Z Ran 6 tests in 0.268s
2023-01-11T21:18:59.5047862Z 
2023-01-11T21:18:59.5047922Z OK (skipped=2)
2023-01-11T21:18:59.5048030Z 
2023-01-11T21:18:59.5048117Z Generating XML reports...
2023-01-11T21:18:59.5048718Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20230111211858.xml
2023-01-11T21:18:59.5049330Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20230111211858.xml
2023-01-11T21:18:59.5049551Z 
2023-01-11T21:18:59.5049819Z ##[endgroup]
2023-01-11T21:18:59.5050255Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_q64npkhd)
2023-01-11T21:18:59.5050513Z 
2023-01-11T21:18:59.5050712Z Running distributed/checkpoint/test_utils ... [2023-01-11 21:18:59.504458]
2023-01-11T21:18:59.5051226Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:18:59.504795]
2023-01-11T21:19:01.3761389Z 
2023-01-11T21:19:01.3762036Z Expand the folded group to see the log file of distributed/checkpoint/test_utils
2023-01-11T21:19:01.3763257Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_utils_di32ngex)
2023-01-11T21:19:01.3764351Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_utils
2023-01-11T21:19:01.3764744Z 
2023-01-11T21:19:01.3764862Z Running tests...
2023-01-11T21:19:01.3765301Z ----------------------------------------------------------------------
2023-01-11T21:19:01.3765663Z   test_flat_data (__main__.TestMedatadaIndex) ... ok (0.272s)
2023-01-11T21:19:01.3765961Z   test_index_hint_ignored_on_equals (__main__.TestMedatadaIndex) ... ok (0.001s)
2023-01-11T21:19:01.3766737Z   test_index_hint_ignored_on_hash (__main__.TestMedatadaIndex) ... ok (0.001s)
2023-01-11T21:19:01.3767121Z   test_init_convert_offset (__main__.TestMedatadaIndex) ... ok (0.001s)
2023-01-11T21:19:01.3767488Z   test_sharded_tensor_lookup (__main__.TestMedatadaIndex) ... ok (0.003s)
2023-01-11T21:19:01.3767659Z 
2023-01-11T21:19:01.3767852Z ----------------------------------------------------------------------
2023-01-11T21:19:01.3768096Z Ran 5 tests in 0.277s
2023-01-11T21:19:01.3768210Z 
2023-01-11T21:19:01.3768306Z OK
2023-01-11T21:19:01.3768399Z 
2023-01-11T21:19:01.3768484Z Generating XML reports...
2023-01-11T21:19:01.3768994Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_utils/TEST-TestMedatadaIndex-20230111211900.xml
2023-01-11T21:19:01.3769324Z 
2023-01-11T21:19:01.3769620Z ##[endgroup]
2023-01-11T21:19:01.3770308Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_utils_di32ngex)
2023-01-11T21:19:01.3770711Z 
2023-01-11T21:19:01.3770912Z Running distributed/fsdp/test_fsdp_fx ... [2023-01-11 21:19:01.376458]
2023-01-11T21:19:01.3771422Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:01.376848]
2023-01-11T21:19:03.2581845Z 
2023-01-11T21:19:03.2582427Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_fx
2023-01-11T21:19:03.2583561Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_xt1g36go)
2023-01-11T21:19:03.2584260Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx
2023-01-11T21:19:03.2584454Z 
2023-01-11T21:19:03.2584533Z Running tests...
2023-01-11T21:19:03.2584849Z ----------------------------------------------------------------------
2023-01-11T21:19:03.2585151Z   test_symbolic_tracing_outputs (__main__.TestSymbolicTracing)
2023-01-11T21:19:03.2585461Z Tests running ``tracer.trace()`` inside ``patch_tracer()`` by checking ... ok (0.284s)
2023-01-11T21:19:03.2585639Z 
2023-01-11T21:19:03.2585844Z ----------------------------------------------------------------------
2023-01-11T21:19:03.2586084Z Ran 1 test in 0.284s
2023-01-11T21:19:03.2586198Z 
2023-01-11T21:19:03.2586266Z OK
2023-01-11T21:19:03.2586374Z 
2023-01-11T21:19:03.2586492Z Generating XML reports...
2023-01-11T21:19:03.2587173Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20230111211902.xml
2023-01-11T21:19:03.2587478Z 
2023-01-11T21:19:03.2587711Z ##[endgroup]
2023-01-11T21:19:03.2588189Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_xt1g36go)
2023-01-11T21:19:03.2588439Z 
2023-01-11T21:19:03.2588677Z Running distributed/checkpoint/test_file_system_checkpoint ... [2023-01-11 21:19:03.258571]
2023-01-11T21:19:03.2591057Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:03.258944]
2023-01-11T21:19:05.1801210Z 
2023-01-11T21:19:05.1802011Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint
2023-01-11T21:19:05.1802979Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_d6_352jl)
2023-01-11T21:19:05.1803698Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint
2023-01-11T21:19:05.1803926Z 
2023-01-11T21:19:05.1804001Z Running tests...
2023-01-11T21:19:05.1804296Z ----------------------------------------------------------------------
2023-01-11T21:19:05.1804727Z   test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:05.1805524Z   test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:05.1806093Z   test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:05.1806578Z   test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:05.1807718Z   test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:19:05.1808546Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:19:05.1808825Z ok (0.297s)
2023-01-11T21:19:05.1809195Z   test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:05.1809483Z 
2023-01-11T21:19:05.1809679Z ----------------------------------------------------------------------
2023-01-11T21:19:05.1809925Z Ran 6 tests in 0.304s
2023-01-11T21:19:05.1810040Z 
2023-01-11T21:19:05.1810114Z OK (skipped=5)
2023-01-11T21:19:05.1810221Z 
2023-01-11T21:19:05.1810306Z Generating XML reports...
2023-01-11T21:19:05.1810829Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20230111211904.xml
2023-01-11T21:19:05.1811541Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20230111211904.xml
2023-01-11T21:19:05.1812316Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20230111211904.xml
2023-01-11T21:19:05.1812691Z 
2023-01-11T21:19:05.1812921Z ##[endgroup]
2023-01-11T21:19:05.1813442Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_d6_352jl)
2023-01-11T21:19:05.1813744Z 
2023-01-11T21:19:05.1813944Z Running distributed/fsdp/test_fsdp_overlap ... [2023-01-11 21:19:05.180426]
2023-01-11T21:19:05.1814471Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:05.180773]
2023-01-11T21:19:08.6952223Z 
2023-01-11T21:19:08.6952841Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_overlap
2023-01-11T21:19:08.6953801Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_fxsy8c3_)
2023-01-11T21:19:08.6954476Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap
2023-01-11T21:19:08.6954685Z 
2023-01-11T21:19:08.6954760Z Running tests...
2023-01-11T21:19:08.6955157Z ----------------------------------------------------------------------
2023-01-11T21:19:08.6955573Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26029
2023-01-11T21:19:08.6956119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:08.6956569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:08.6957327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:08.6957687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:08.6958257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:19:08.6958912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:19:08.6959310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:08.6973103Z dist init r=0, world=1
2023-01-11T21:19:08.6973664Z skip: Need at least 2 CUDA devices (1.870s)
2023-01-11T21:19:08.6974718Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2023-01-11T21:19:08.6975188Z 
2023-01-11T21:19:08.6975401Z ----------------------------------------------------------------------
2023-01-11T21:19:08.6975634Z Ran 2 tests in 1.871s
2023-01-11T21:19:08.6975748Z 
2023-01-11T21:19:08.6975823Z OK (skipped=2)
2023-01-11T21:19:08.6975936Z 
2023-01-11T21:19:08.6976020Z Generating XML reports...
2023-01-11T21:19:08.6976511Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20230111211906.xml
2023-01-11T21:19:08.6977180Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20230111211906.xml
2023-01-11T21:19:08.6977473Z 
2023-01-11T21:19:08.6977767Z ##[endgroup]
2023-01-11T21:19:08.6978222Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_fxsy8c3_)
2023-01-11T21:19:08.6978473Z 
2023-01-11T21:19:08.6978714Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2023-01-11 21:19:08.695437]
2023-01-11T21:19:08.6979282Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:08.695807]
2023-01-11T21:19:12.3823804Z 
2023-01-11T21:19:12.3825400Z Expand the folded group to see the log file of distributed/_shard/sharding_spec/test_sharding_spec
2023-01-11T21:19:12.3826469Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_1tt94g8b)
2023-01-11T21:19:12.3827432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjg2hgd92
2023-01-11T21:19:12.3828526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjg2hgd92/_remote_module_non_scriptable.py
2023-01-11T21:19:12.3829780Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec
2023-01-11T21:19:12.3830191Z 
2023-01-11T21:19:12.3830330Z Running tests...
2023-01-11T21:19:12.3830915Z ----------------------------------------------------------------------
2023-01-11T21:19:12.3831688Z   test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26054
2023-01-11T21:19:12.3832493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26055
2023-01-11T21:19:12.3833230Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26056
2023-01-11T21:19:12.3833952Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26057
2023-01-11T21:19:12.3834996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:12.3835709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:12.3836673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:12.3837759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:12.3838624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1msheju
2023-01-11T21:19:12.3839489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1msheju/_remote_module_non_scriptable.py
2023-01-11T21:19:12.3840337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:12.3841389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:12.3842164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:12.3843130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:12.3843880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:12.3844653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbf6gssue
2023-01-11T21:19:12.3845543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbf6gssue/_remote_module_non_scriptable.py
2023-01-11T21:19:12.3846381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:19:12.3847392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:12.3848107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:12.3849036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:12.3849793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:12.3850538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7a0scos
2023-01-11T21:19:12.3851391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7a0scos/_remote_module_non_scriptable.py
2023-01-11T21:19:12.3852468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:12.3853188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:12.3854141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:12.3854866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:12.3855617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyz1e4tc6
2023-01-11T21:19:12.3856497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyz1e4tc6/_remote_module_non_scriptable.py
2023-01-11T21:19:12.3862747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:19:12.3863436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:19:12.3863940Z ok (1.987s)
2023-01-11T21:19:12.3864357Z   test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec)
2023-01-11T21:19:12.3864746Z Test custom spec can be invoked from the ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:12.3865088Z   test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec)
2023-01-11T21:19:12.3865440Z Test sharded_tensor.ones(...) with the custom ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:12.3865770Z   test_check_overlapping (__main__.TestShardingSpec) ... ok (0.003s)
2023-01-11T21:19:12.3866085Z   test_chunked_sharding_spec (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.001s)
2023-01-11T21:19:12.3866426Z   test_device_placement (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.001s)
2023-01-11T21:19:12.3866774Z   test_enumerable_sharding_spec (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.003s)
2023-01-11T21:19:12.3867216Z   test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s)
2023-01-11T21:19:12.3867517Z   test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s)
2023-01-11T21:19:12.3867858Z   test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s)
2023-01-11T21:19:12.3868167Z   test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.007s)
2023-01-11T21:19:12.3868410Z 
2023-01-11T21:19:12.3868679Z ----------------------------------------------------------------------
2023-01-11T21:19:12.3868926Z Ran 11 tests in 2.008s
2023-01-11T21:19:12.3869042Z 
2023-01-11T21:19:12.3869118Z OK (skipped=5)
2023-01-11T21:19:12.3869225Z 
2023-01-11T21:19:12.3869297Z Generating XML reports...
2023-01-11T21:19:12.3869811Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20230111211909.xml
2023-01-11T21:19:12.3870456Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20230111211909.xml
2023-01-11T21:19:12.3870727Z 
2023-01-11T21:19:12.3871079Z ##[endgroup]
2023-01-11T21:19:12.3871587Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_1tt94g8b)
2023-01-11T21:19:12.3871884Z 
2023-01-11T21:19:12.3872104Z Running distributed/checkpoint/test_fsdp_optim_state ... [2023-01-11 21:19:12.382704]
2023-01-11T21:19:12.3872651Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:12.383159]
2023-01-11T21:19:16.1505623Z 
2023-01-11T21:19:16.1506280Z Expand the folded group to see the log file of distributed/checkpoint/test_fsdp_optim_state
2023-01-11T21:19:16.1507725Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_fsdp_optim_state_u26j5ipp)
2023-01-11T21:19:16.1508577Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_fsdp_optim_state
2023-01-11T21:19:16.1508805Z 
2023-01-11T21:19:16.1508892Z Running tests...
2023-01-11T21:19:16.1509292Z ----------------------------------------------------------------------
2023-01-11T21:19:16.1509921Z   test_distributed_tensor_planner (__main__.FsdpOptimStateCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26109
2023-01-11T21:19:16.1510398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26110
2023-01-11T21:19:16.1510895Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26111
2023-01-11T21:19:16.1511442Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26112
2023-01-11T21:19:16.1512293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:16.1512816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:16.1513594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:16.1514231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:16.1514838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:19:16.1515408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:16.1515731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:16.1516159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:16.1516506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:16.1516817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:19:16.1517538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:16.1517941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:16.1518380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:16.1518715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:16.1519041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:19:16.1519501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:16.1519837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:16.1520253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:16.1520602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:16.1520934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:16.1521284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:19:16.1521750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:19:16.1522120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:19:16.1522487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:19:16.1522973Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:16.1523499Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:16.1524025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:16.1524544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:16.1524859Z skip: Need at least 4 CUDA devices (2.085s)
2023-01-11T21:19:16.1524996Z 
2023-01-11T21:19:16.1525202Z ----------------------------------------------------------------------
2023-01-11T21:19:16.1525447Z Ran 1 test in 2.085s
2023-01-11T21:19:16.1525561Z 
2023-01-11T21:19:16.1525633Z OK (skipped=1)
2023-01-11T21:19:16.1525729Z 
2023-01-11T21:19:16.1525814Z Generating XML reports...
2023-01-11T21:19:16.1526322Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_fsdp_optim_state/TEST-FsdpOptimStateCheckpoint-20230111211913.xml
2023-01-11T21:19:16.1526618Z 
2023-01-11T21:19:16.1526905Z ##[endgroup]
2023-01-11T21:19:16.1527392Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_fsdp_optim_state_u26j5ipp)
2023-01-11T21:19:16.1527681Z 
2023-01-11T21:19:16.1527882Z Running distributed/fsdp/test_fsdp_uneven ... [2023-01-11 21:19:16.150864]
2023-01-11T21:19:16.1542626Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:16.151286]
2023-01-11T21:19:19.9373268Z 
2023-01-11T21:19:19.9373966Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_uneven
2023-01-11T21:19:19.9375054Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_uneven_25bu2rw1)
2023-01-11T21:19:19.9376023Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven
2023-01-11T21:19:19.9376376Z 
2023-01-11T21:19:19.9376516Z Running tests...
2023-01-11T21:19:19.9377304Z ----------------------------------------------------------------------
2023-01-11T21:19:19.9378397Z   test_one_iteration (__main__.TestUnevenParamShard)
2023-01-11T21:19:19.9379187Z Test FSDP with uneven divide of parameter shards. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26176
2023-01-11T21:19:19.9379845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26177
2023-01-11T21:19:19.9380373Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26178
2023-01-11T21:19:19.9380972Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26179
2023-01-11T21:19:19.9382071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:19.9382602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:19.9383031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:19.9383453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:19.9383796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:19:19.9384319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:19.9384657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:19.9385153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:19.9385537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:19.9385873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:19:19.9386407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:19.9386739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:19.9387226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:19.9387573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:19.9387968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:19:19.9388470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:19.9389027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:19.9389361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:19.9389854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:19.9390217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:19.9390588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:19:19.9391152Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:19.9391678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:19.9392250Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:19.9392720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:19.9393073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:19:19.9393469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:19:19.9393917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:19:19.9394202Z dist init r=0, world=4
2023-01-11T21:19:19.9394476Z dist init r=3, world=4
2023-01-11T21:19:19.9394660Z dist init r=2, world=4
2023-01-11T21:19:19.9394821Z dist init r=1, world=4
2023-01-11T21:19:19.9395082Z skip: Need at least 2 CUDA devices (2.089s)
2023-01-11T21:19:19.9395223Z 
2023-01-11T21:19:19.9395430Z ----------------------------------------------------------------------
2023-01-11T21:19:19.9395662Z Ran 1 test in 2.089s
2023-01-11T21:19:19.9395804Z 
2023-01-11T21:19:19.9395906Z OK (skipped=1)
2023-01-11T21:19:19.9396019Z 
2023-01-11T21:19:19.9396104Z Generating XML reports...
2023-01-11T21:19:19.9396586Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20230111211917.xml
2023-01-11T21:19:19.9396881Z 
2023-01-11T21:19:19.9397182Z ##[endgroup]
2023-01-11T21:19:19.9397705Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_uneven_25bu2rw1)
2023-01-11T21:19:19.9397971Z 
2023-01-11T21:19:19.9398258Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2023-01-11 21:19:19.937635]
2023-01-11T21:19:19.9398803Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:19.938260]
2023-01-11T21:19:23.6754988Z 
2023-01-11T21:19:23.6756076Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_wrapping
2023-01-11T21:19:23.6758433Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_a5aovj53)
2023-01-11T21:19:23.6759588Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping
2023-01-11T21:19:23.6760107Z 
2023-01-11T21:19:23.6760228Z Running tests...
2023-01-11T21:19:23.6760671Z ----------------------------------------------------------------------
2023-01-11T21:19:23.6761062Z   test_multiple_wrapping (__main__.TestMultipleWrapping)
2023-01-11T21:19:23.6761576Z This test simulates wrapping the module after training to run inference. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26243
2023-01-11T21:19:23.6761993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26244
2023-01-11T21:19:23.6762391Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26245
2023-01-11T21:19:23.6762708Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26246
2023-01-11T21:19:23.6763247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:23.6763585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:23.6764084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:23.6764472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:23.6764943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:19:23.6765557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:23.6765879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:23.6766365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:23.6766720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:23.6767119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:19:23.6767581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:23.6768157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:23.6768718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:23.6769072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:23.6769502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:19:23.6769984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:23.6770375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:23.6770791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:23.6771189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:23.6771529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:19:23.6772091Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:23.6772659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:23.6773225Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:23.6773754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:23.6774194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:19:23.6774528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:19:23.6774920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:23.6775265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:19:23.6775562Z dist init r=3, world=4
2023-01-11T21:19:23.6775744Z dist init r=2, world=4
2023-01-11T21:19:23.6775925Z dist init r=0, world=4
2023-01-11T21:19:23.6776090Z dist init r=1, world=4
2023-01-11T21:19:23.6776344Z skip: Need at least 2 CUDA devices (2.080s)
2023-01-11T21:19:23.6776485Z 
2023-01-11T21:19:23.6776688Z ----------------------------------------------------------------------
2023-01-11T21:19:23.6776944Z Ran 1 test in 2.080s
2023-01-11T21:19:23.6777080Z 
2023-01-11T21:19:23.6777152Z OK (skipped=1)
2023-01-11T21:19:23.6777258Z 
2023-01-11T21:19:23.6777342Z Generating XML reports...
2023-01-11T21:19:23.6777877Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20230111211921.xml
2023-01-11T21:19:23.6778167Z 
2023-01-11T21:19:23.6778429Z ##[endgroup]
2023-01-11T21:19:23.6778977Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_a5aovj53)
2023-01-11T21:19:23.6779297Z 
2023-01-11T21:19:23.6779546Z Running distributed/fsdp/test_fsdp_input ... [2023-01-11 21:19:23.675875]
2023-01-11T21:19:23.6780073Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:23.676389]
2023-01-11T21:19:28.8116083Z 
2023-01-11T21:19:28.8116630Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_input
2023-01-11T21:19:28.8117416Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_6la8hrv8)
2023-01-11T21:19:28.8118116Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input
2023-01-11T21:19:28.8118565Z 
2023-01-11T21:19:28.8118628Z Running tests...
2023-01-11T21:19:28.8118998Z ----------------------------------------------------------------------
2023-01-11T21:19:28.8119341Z   test_input_type_dict (__main__.TestInput)
2023-01-11T21:19:28.8119749Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26310
2023-01-11T21:19:28.8120302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:28.8120654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:28.8121248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:28.8121773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:28.8122169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:19:28.8122725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:19:28.8123130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:28.8123379Z dist init r=0, world=1
2023-01-11T21:19:28.8123643Z skip: Need at least 1 CUDA device (1.867s)
2023-01-11T21:19:28.8123908Z   test_input_type_list (__main__.TestInput)
2023-01-11T21:19:28.8124323Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26324
2023-01-11T21:19:28.8124829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:28.8125222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:28.8125756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:28.8126231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:28.8126596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:19:28.8127125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:19:28.8127567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:28.8127815Z dist init r=0, world=1
2023-01-11T21:19:28.8128021Z skip: Need at least 1 CUDA device (1.609s)
2023-01-11T21:19:28.8128213Z 
2023-01-11T21:19:28.8128419Z ----------------------------------------------------------------------
2023-01-11T21:19:28.8128719Z Ran 2 tests in 3.476s
2023-01-11T21:19:28.8128819Z 
2023-01-11T21:19:28.8128924Z OK (skipped=2)
2023-01-11T21:19:28.8129058Z 
2023-01-11T21:19:28.8129143Z Generating XML reports...
2023-01-11T21:19:28.8129577Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20230111211924.xml
2023-01-11T21:19:28.8129885Z 
2023-01-11T21:19:28.8130133Z ##[endgroup]
2023-01-11T21:19:28.8130652Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_6la8hrv8)
2023-01-11T21:19:28.8130911Z 
2023-01-11T21:19:28.8131120Z Running distributed/test_c10d_error_logger ... [2023-01-11 21:19:28.811980]
2023-01-11T21:19:28.8131705Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_error_logger.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:28.812452]
2023-01-11T21:19:33.9519193Z 
2023-01-11T21:19:33.9519963Z Expand the folded group to see the log file of distributed/test_c10d_error_logger
2023-01-11T21:19:33.9521032Z ##[group]PRINTING LOG FILE of distributed/test_c10d_error_logger (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_error_logger_nznxhm7t)
2023-01-11T21:19:33.9521983Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_error_logger
2023-01-11T21:19:33.9522248Z 
2023-01-11T21:19:33.9522351Z Running tests...
2023-01-11T21:19:33.9522819Z ----------------------------------------------------------------------
2023-01-11T21:19:33.9523389Z   test_exception_handler_with_dist (__main__.C10dErrorLoggerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26349
2023-01-11T21:19:33.9523798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26350
2023-01-11T21:19:33.9524311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:33.9524640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:33.9525131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:33.9525482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:33.9525838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:33.9526357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:33.9526797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:33.9527251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:33.9527616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:33.9527939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:19:33.9528296Z skip: Need at least 2 CUDA devices (1.874s)
2023-01-11T21:19:33.9528653Z   test_get_or_create_logger (__main__.C10dErrorLoggerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26371
2023-01-11T21:19:33.9529097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26372
2023-01-11T21:19:33.9529570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:33.9529941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:33.9530399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:33.9530759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:33.9531083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:19:33.9531604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:33.9531965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:33.9532407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:33.9532812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:33.9533132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:33.9533390Z ok (1.613s)
2023-01-11T21:19:33.9533548Z 
2023-01-11T21:19:33.9533759Z ----------------------------------------------------------------------
2023-01-11T21:19:33.9534006Z Ran 2 tests in 3.487s
2023-01-11T21:19:33.9534108Z 
2023-01-11T21:19:33.9534181Z OK (skipped=1)
2023-01-11T21:19:33.9534342Z 
2023-01-11T21:19:33.9534429Z Generating XML reports...
2023-01-11T21:19:33.9534881Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_error_logger/TEST-C10dErrorLoggerTest-20230111211930.xml
2023-01-11T21:19:33.9535203Z 
2023-01-11T21:19:33.9535455Z ##[endgroup]
2023-01-11T21:19:33.9535973Z FINISHED PRINTING LOG FILE of distributed/test_c10d_error_logger (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_error_logger_nznxhm7t)
2023-01-11T21:19:33.9536314Z 
2023-01-11T21:19:33.9536530Z Running distributed/fsdp/test_fsdp_memory ... [2023-01-11 21:19:33.952183]
2023-01-11T21:19:33.9537146Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:33.952553]
2023-01-11T21:19:39.3307886Z 
2023-01-11T21:19:39.3308482Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_memory
2023-01-11T21:19:39.3309630Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_bsinee8l)
2023-01-11T21:19:39.3311903Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory
2023-01-11T21:19:39.3312183Z 
2023-01-11T21:19:39.3312262Z Running tests...
2023-01-11T21:19:39.3312592Z ----------------------------------------------------------------------
2023-01-11T21:19:39.3313003Z   test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26404
2023-01-11T21:19:39.3313387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26405
2023-01-11T21:19:39.3313922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:39.3314312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:39.3314930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:39.3315310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:39.3315650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:19:39.3316130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:39.3316470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:39.3316892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:39.3317244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:39.3317579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:19:39.3318062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:19:39.3318642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:19:39.3319151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:39.3319501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:19:39.3319751Z dist init r=0, world=2
2023-01-11T21:19:39.3319932Z dist init r=1, world=2
2023-01-11T21:19:39.3320141Z skip: Need at least 2 CUDA devices (1.978s)
2023-01-11T21:19:39.3320484Z   test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26432
2023-01-11T21:19:39.3320875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26433
2023-01-11T21:19:39.3321332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:39.3321757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:39.3322177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:39.3322529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:39.3322867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:19:39.3323539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:39.3323938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:39.3324374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:39.3324724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:39.3325045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:19:39.3325542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:19:39.3326089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:19:39.3326476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:39.3326810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:19:39.3327067Z dist init r=0, world=2
2023-01-11T21:19:39.3327249Z dist init r=1, world=2
2023-01-11T21:19:39.3327448Z skip: Need at least 2 CUDA devices (1.714s)
2023-01-11T21:19:39.3327584Z 
2023-01-11T21:19:39.3327787Z ----------------------------------------------------------------------
2023-01-11T21:19:39.3328033Z Ran 2 tests in 3.693s
2023-01-11T21:19:39.3328146Z 
2023-01-11T21:19:39.3328218Z OK (skipped=2)
2023-01-11T21:19:39.3328313Z 
2023-01-11T21:19:39.3328397Z Generating XML reports...
2023-01-11T21:19:39.3328850Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20230111211935.xml
2023-01-11T21:19:39.3329104Z 
2023-01-11T21:19:39.3329386Z ##[endgroup]
2023-01-11T21:19:39.3329822Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_bsinee8l)
2023-01-11T21:19:39.3330087Z 
2023-01-11T21:19:39.3330299Z Running distributed/elastic/timer/local_timer_test ... [2023-01-11 21:19:39.331332]
2023-01-11T21:19:39.3330839Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:39.331781]
2023-01-11T21:19:44.8115764Z 
2023-01-11T21:19:44.8116309Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_test
2023-01-11T21:19:44.8117704Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_test_u8ei0e1j)
2023-01-11T21:19:44.8118779Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_test
2023-01-11T21:19:44.8119048Z 
2023-01-11T21:19:44.8119170Z Running tests...
2023-01-11T21:19:44.8119648Z ----------------------------------------------------------------------
2023-01-11T21:19:44.8120004Z   test_acquire_release (__main__.LocalTimerServerTest)
2023-01-11T21:19:44.8120245Z tests that: ... ok (0.259s)
2023-01-11T21:19:44.8120469Z   test_expired_timers (__main__.LocalTimerServerTest)
2023-01-11T21:19:44.8120853Z tests that a single expired timer on a process should terminate ... ok (0.002s)
2023-01-11T21:19:44.8121152Z   test_valid_timers (__main__.LocalTimerServerTest)
2023-01-11T21:19:44.8121592Z tests that valid timers are processed correctly and the process is left alone ... ok (0.002s)
2023-01-11T21:19:44.8121896Z   test_watchdog_call_count (__main__.LocalTimerServerTest)
2023-01-11T21:19:44.8122299Z checks that the watchdog function ran wait/interval +- 1 times ... ok (0.102s)
2023-01-11T21:19:44.8122771Z   test_watchdog_empty_queue (__main__.LocalTimerServerTest)
2023-01-11T21:19:44.8123115Z checks that the watchdog can run on an empty queue ... ok (0.011s)
2023-01-11T21:19:44.8123707Z   test_client_interaction (__main__.LocalTimerTest) ... ok (0.007s)
2023-01-11T21:19:44.8124017Z   test_exception_propagation (__main__.LocalTimerTest) ... ok (0.011s)
2023-01-11T21:19:44.8124401Z   test_get_timer_recursive (__main__.LocalTimerTest)
2023-01-11T21:19:44.8124678Z If a function acquires a countdown timer with default scope, ... ok (1.697s)
2023-01-11T21:19:44.8125034Z   test_happy_path (__main__.LocalTimerTest) ... ok (0.103s)
2023-01-11T21:19:44.8125298Z   test_no_client (__main__.LocalTimerTest) ... ok (0.011s)
2023-01-11T21:19:44.8125563Z   test_timer (__main__.LocalTimerTest) ... ok (0.130s)
2023-01-11T21:19:44.8125880Z   test_get (__main__.MultiprocessingRequestQueueTest) ... ok (0.022s)
2023-01-11T21:19:44.8126203Z   test_get_less_than_size (__main__.MultiprocessingRequestQueueTest)
2023-01-11T21:19:44.8126514Z Tests slow producer. ... ok (0.507s)
2023-01-11T21:19:44.8126775Z   test_get_size (__main__.MultiprocessingRequestQueueTest)
2023-01-11T21:19:44.8127086Z Creates a "producer" process that enqueues ``n`` elements ... ok (0.910s)
2023-01-11T21:19:44.8127304Z 
2023-01-11T21:19:44.8127520Z ----------------------------------------------------------------------
2023-01-11T21:19:44.8127754Z Ran 14 tests in 3.777s
2023-01-11T21:19:44.8127900Z 
2023-01-11T21:19:44.8127994Z OK
2023-01-11T21:19:44.8128089Z 
2023-01-11T21:19:44.8128175Z Generating XML reports...
2023-01-11T21:19:44.8128680Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20230111211940.xml
2023-01-11T21:19:44.8129333Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20230111211940.xml
2023-01-11T21:19:44.8130064Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20230111211940.xml
2023-01-11T21:19:44.8130440Z 
2023-01-11T21:19:44.8130709Z ##[endgroup]
2023-01-11T21:19:44.8131251Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_test_u8ei0e1j)
2023-01-11T21:19:44.8131542Z 
2023-01-11T21:19:44.8131825Z Running distributed/_shard/sharded_tensor/test_sharded_tensor ... [2023-01-11 21:19:44.811787]
2023-01-11T21:19:44.8132407Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:44.812172]
2023-01-11T21:19:50.3878417Z 
2023-01-11T21:19:50.3879055Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor
2023-01-11T21:19:50.3880317Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_mntu7oqk)
2023-01-11T21:19:50.3881321Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor
2023-01-11T21:19:50.3881622Z 
2023-01-11T21:19:50.3881700Z Running tests...
2023-01-11T21:19:50.3882001Z ----------------------------------------------------------------------
2023-01-11T21:19:50.3882348Z   test_empty (__main__.TestCreateTensorFromParams) ... skip: CUDA GPU is needed (0.001s)
2023-01-11T21:19:50.3882716Z   test_local_tensor (__main__.TestLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3883077Z   test_local_tensor_error (__main__.TestLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:19:50.3883609Z   test_collect_local_shard (__main__.TestModuleHookApi) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:19:50.3884121Z   test_reshard_output (__main__.TestModuleHookApi) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3884512Z   test_create_shard_with_no_placement (__main__.TestShardMetadata) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:19:50.3885231Z   test_shard_metadata_init (__main__.TestShardMetadata) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3885698Z   test_shard_parameter (__main__.TestShardParameter) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3886166Z   test_shard_parameter_errors (__main__.TestShardParameter) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3886555Z   test_shard_tensor (__main__.TestShardTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3886990Z   test_shard_tensor_errors (__main__.TestShardTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3887571Z   test_cleanup (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:19:50.3888022Z   test_complete_world_size (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3888448Z   test_create_sharded_tensor_like (__main__.TestShardedTensorChunked)
2023-01-11T21:19:50.3888805Z Test tensor like methods, i.e. torch.zeros_like(...), torch.full_like, etc. ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3889230Z   test_create_sharded_tensor_with_full (__main__.TestShardedTensorChunked)
2023-01-11T21:19:50.3889558Z Test sharded_tensor.full(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3889923Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorChunked)
2023-01-11T21:19:50.3890250Z Test sharded_tensor.ones(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3890623Z   test_create_sharded_tensor_with_rand (__main__.TestShardedTensorChunked)
2023-01-11T21:19:50.3890968Z Test sharded_tensor.rand(...)/randn(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3891299Z   test_create_sharded_tensor_with_zeros (__main__.TestShardedTensorChunked)
2023-01-11T21:19:50.3891679Z Test sharded_tensor.zeros(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3891985Z   test_gather_even (__main__.TestShardedTensorChunked)
2023-01-11T21:19:50.3892376Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3892719Z   test_gather_uneven (__main__.TestShardedTensorChunked)
2023-01-11T21:19:50.3893125Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3893548Z   test_insufficient_sharding_dims (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3894019Z   test_invalid_pg_rpc_ranks (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3894468Z   test_invalid_sharding (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3894904Z   test_load_state_dict_errors (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3895385Z   test_multiple_local_shards (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3895774Z   test_new_group (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3896228Z   test_partial_world_size (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3896639Z   test_sharded_tensor_metadata (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3897114Z   test_sharded_tensor_sizes (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3897637Z   test_sharding_columns (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3898084Z   test_state_dict (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3898543Z   test_state_dict_new_group (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3898965Z   test_state_dict_no_sharded_tensors (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3899423Z   test_custom_op (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3899837Z   test_custom_op_errors (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:19:50.3900294Z   test_custom_op_override (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3900713Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorEnumerable)
2023-01-11T21:19:50.3901047Z Test sharded_tensor.ones(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3901359Z   test_gather_even (__main__.TestShardedTensorEnumerable)
2023-01-11T21:19:50.3902075Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3902486Z   test_gather_uneven (__main__.TestShardedTensorEnumerable)
2023-01-11T21:19:50.3902848Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3903322Z   test_grid_sharding (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3903768Z   test_multiple_local_shards (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3904203Z   test_new_group (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3904671Z   test_partial_world_size (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3905099Z   test_sharded_tensor_device (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3905584Z   test_sharded_tensor_metadata (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3906002Z   test_sharded_tensor_to_cpu (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3906485Z   test_sharded_tensor_to_cuda (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3906966Z   test_sharded_tensor_to_test (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3907388Z   test_uneven_shards (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3907896Z   test_with_rpc_names (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3908316Z   test_init_from_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3908851Z   test_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3909415Z   test_init_from_local_shards_and_global_metadata_invalid_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3910066Z   test_init_from_local_shards_invalid_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3910646Z   test_init_from_local_shards_invalid_pin_memory (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26514
2023-01-11T21:19:50.3911141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26515
2023-01-11T21:19:50.3911489Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26516
2023-01-11T21:19:50.3911866Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26517
2023-01-11T21:19:50.3912446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:50.3912789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:50.3913286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:50.3913641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:50.3914011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:19:50.3914485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:50.3914879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:50.3915312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:50.3915710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:50.3916044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:19:50.3916572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:50.3916899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:50.3917383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:50.3917729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:50.3918108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:19:50.3918565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:50.3918893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:50.3919368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:50.3919710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:50.3920035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:50.3920399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:19:50.3920771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:19:50.3921122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:19:50.3921588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:19:50.3922109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:50.3922636Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:50.3923150Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:50.3923725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:19:50.3924085Z skip: Need at least 4 CUDA devices (2.079s)
2023-01-11T21:19:50.3924457Z   test_init_from_local_shards_invalid_property_cross_ranks (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:19:50.3924958Z   test_init_from_local_shards_invalid_shards_gaps (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3925524Z   test_init_from_local_shards_invalid_shards_overlap (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3926008Z   test_init_from_local_shards_new_group (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3926460Z   test_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3926948Z   test_st_base_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26570
2023-01-11T21:19:50.3927393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26571
2023-01-11T21:19:50.3927727Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26572
2023-01-11T21:19:50.3928058Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26573
2023-01-11T21:19:50.3928508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:50.3928839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:50.3929270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:50.3929607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:50.3929932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:19:50.3930393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:50.3930725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:50.3931140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:50.3931483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:50.3931808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:19:50.3932260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:50.3932646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:50.3933078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:50.3933424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:50.3933733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:50.3934192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:50.3934518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:50.3934948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:50.3935280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:50.3935602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:19:50.3935954Z skip: Need at least 4 CUDA devices (1.720s)
2023-01-11T21:19:50.3936345Z   test_init_from_local_tensor (__main__.TestShardedTensorFromLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3936809Z   test_init_from_local_tensor_errors (__main__.TestShardedTensorFromLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:19:50.3937210Z   test_serialize_and_deserialize (__main__.TestShardedTensorMetadata) ... ok (0.033s)
2023-01-11T21:19:50.3937400Z 
2023-01-11T21:19:50.3937601Z ----------------------------------------------------------------------
2023-01-11T21:19:50.3937832Z Ran 64 tests in 3.902s
2023-01-11T21:19:50.3937947Z 
2023-01-11T21:19:50.3938025Z OK (skipped=63)
2023-01-11T21:19:50.3938135Z 
2023-01-11T21:19:50.3938220Z Generating XML reports...
2023-01-11T21:19:50.3938719Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20230111211946.xml
2023-01-11T21:19:50.3939399Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20230111211946.xml
2023-01-11T21:19:50.3940042Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20230111211946.xml
2023-01-11T21:19:50.3940661Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20230111211946.xml
2023-01-11T21:19:50.3941281Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20230111211946.xml
2023-01-11T21:19:50.3942040Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20230111211946.xml
2023-01-11T21:19:50.3942661Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20230111211946.xml
2023-01-11T21:19:50.3943301Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20230111211946.xml
2023-01-11T21:19:50.3943972Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20230111211946.xml
2023-01-11T21:19:50.3944631Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20230111211946.xml
2023-01-11T21:19:50.3945321Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20230111211946.xml
2023-01-11T21:19:50.3946083Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20230111211946.xml
2023-01-11T21:19:50.3946403Z 
2023-01-11T21:19:50.3946716Z ##[endgroup]
2023-01-11T21:19:50.3947240Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_mntu7oqk)
2023-01-11T21:19:50.3947537Z 
2023-01-11T21:19:50.3947737Z Running distributed/fsdp/test_fsdp_apply ... [2023-01-11 21:19:50.388263]
2023-01-11T21:19:50.3948254Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:50.388624]
2023-01-11T21:19:57.4576894Z 
2023-01-11T21:19:57.4577438Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_apply
2023-01-11T21:19:57.4578558Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_hv2zzfx3)
2023-01-11T21:19:57.4579390Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply
2023-01-11T21:19:57.4579935Z 
2023-01-11T21:19:57.4580074Z Running tests...
2023-01-11T21:19:57.4580588Z ----------------------------------------------------------------------
2023-01-11T21:19:57.4580945Z   test_apply_in_summon_raises_error (__main__.TestApply)
2023-01-11T21:19:57.4581316Z Tests that calling ``apply()`` on an FSDP instance inside the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26625
2023-01-11T21:19:57.4581976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26626
2023-01-11T21:19:57.4582628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:57.4583061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:57.4583644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:57.4584028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:57.4584352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:19:57.4584831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:57.4585164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:57.4585593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:57.4585987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:57.4586379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:19:57.4587001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:19:57.4587600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:19:57.4587981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:19:57.4588338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:57.4588600Z dist init r=0, world=2
2023-01-11T21:19:57.4588770Z dist init r=1, world=2
2023-01-11T21:19:57.4588981Z skip: Need at least 2 CUDA devices (1.976s)
2023-01-11T21:19:57.4589223Z   test_nested_module_apply (__main__.TestApply)
2023-01-11T21:19:57.4589680Z Tests that ``apply()`` modifies parameter values in-place on a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26653
2023-01-11T21:19:57.4590080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26654
2023-01-11T21:19:57.4590541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:57.4590877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:57.4591297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:57.4591643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:57.4591982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:19:57.4592456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:57.4592771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:57.4593198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:57.4593542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:57.4593862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:19:57.4594460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:19:57.4595056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:19:57.4595445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:19:57.4595778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:57.4596033Z dist init r=1, world=2
2023-01-11T21:19:57.4596215Z dist init r=0, world=2
2023-01-11T21:19:57.4596411Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:19:57.4596657Z   test_transformer_module_apply (__main__.TestApply)
2023-01-11T21:19:57.4597141Z Tests that ``apply()`` modifies parameter values in-place on an ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26681
2023-01-11T21:19:57.4597545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26682
2023-01-11T21:19:57.4597990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:57.4598319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:57.4598755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:57.4599087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:57.4599424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:19:57.4599896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:19:57.4606029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:19:57.4607032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:19:57.4607422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:19:57.4607771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:19:57.4608265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:19:57.4608796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:19:57.4609406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:19:57.4610056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:19:57.4610312Z dist init r=0, world=2
2023-01-11T21:19:57.4610495Z dist init r=1, world=2
2023-01-11T21:19:57.4610706Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:19:57.4610849Z 
2023-01-11T21:19:57.4611045Z ----------------------------------------------------------------------
2023-01-11T21:19:57.4611291Z Ran 3 tests in 5.402s
2023-01-11T21:19:57.4611406Z 
2023-01-11T21:19:57.4611494Z OK (skipped=3)
2023-01-11T21:19:57.4611656Z 
2023-01-11T21:19:57.4611802Z Generating XML reports...
2023-01-11T21:19:57.4612482Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20230111211951.xml
2023-01-11T21:19:57.4612868Z 
2023-01-11T21:19:57.4613335Z ##[endgroup]
2023-01-11T21:19:57.4613802Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_hv2zzfx3)
2023-01-11T21:19:57.4614067Z 
2023-01-11T21:19:57.4614266Z Running distributed/_tensor/test_math_ops ... [2023-01-11 21:19:57.458123]
2023-01-11T21:19:57.4614773Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:19:57.458555]
2023-01-11T21:20:04.7226134Z 
2023-01-11T21:20:04.7227076Z Expand the folded group to see the log file of distributed/_tensor/test_math_ops
2023-01-11T21:20:04.7228418Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_math_ops_v8hr1mbo)
2023-01-11T21:20:04.7229340Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_math_ops
2023-01-11T21:20:04.7229640Z 
2023-01-11T21:20:04.7229753Z Running tests...
2023-01-11T21:20:04.7230263Z ----------------------------------------------------------------------
2023-01-11T21:20:04.7230904Z   test_softmax_fwd (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26720
2023-01-11T21:20:04.7231609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26721
2023-01-11T21:20:04.7232113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26722
2023-01-11T21:20:04.7232695Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26723
2023-01-11T21:20:04.7233563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:04.7234062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:04.7234767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:04.7235319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:04.7235888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:04.7236612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:04.7236985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:04.7237558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:04.7238031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:04.7238375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:04.7238849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:04.7239189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:04.7239611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:04.7239962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:04.7240293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:04.7240754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:04.7241079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:04.7241594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:04.7241946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:04.7242263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:04.7242631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:04.7243005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:04.7243373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:04.7243732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:04.7244395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:04.7244969Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:04.7245498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:04.7246002Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:04.7246300Z ok (2.087s)
2023-01-11T21:20:04.7246619Z   test_softmax_with_bwd (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26776
2023-01-11T21:20:04.7247040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26777
2023-01-11T21:20:04.7247361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26778
2023-01-11T21:20:04.7247688Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26779
2023-01-11T21:20:04.7248148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:04.7248472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:04.7248909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:04.7249258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:04.7249585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:04.7250040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:04.7250374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:04.7250810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:04.7251165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:04.7251480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:04.7251943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:04.7252274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:04.7252693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:04.7253041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:04.7253368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:04.7253832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:04.7254153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:04.7254587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:04.7254936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:04.7255248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:04.7255614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:04.7255982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:04.7256348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:04.7256709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:04.7257244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:04.7257789Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:04.7258318Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:04.7258840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:04.7259168Z skip: Need at least 4 CUDA devices (1.722s)
2023-01-11T21:20:04.7259491Z   test_sum (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26832
2023-01-11T21:20:04.7259869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26833
2023-01-11T21:20:04.7260202Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26834
2023-01-11T21:20:04.7260522Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26835
2023-01-11T21:20:04.7260987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:04.7261323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:04.7261978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:04.7262505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:04.7262918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:04.7263662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:04.7264181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:04.7264975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:04.7265334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:04.7265660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:04.7266117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:04.7266460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:04.7266895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:04.7267244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:04.7267558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:04.7268025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:04.7268358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:04.7268782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:04.7269128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:04.7269452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:04.7269817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:04.7270171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:04.7270540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:04.7270899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:04.7271507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:04.7272069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:04.7272605Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:04.7273127Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:04.7273419Z ok (1.820s)
2023-01-11T21:20:04.7273526Z 
2023-01-11T21:20:04.7273722Z ----------------------------------------------------------------------
2023-01-11T21:20:04.7273969Z Ran 3 tests in 5.629s
2023-01-11T21:20:04.7274084Z 
2023-01-11T21:20:04.7274158Z OK (skipped=1)
2023-01-11T21:20:04.7274270Z 
2023-01-11T21:20:04.7274346Z Generating XML reports...
2023-01-11T21:20:04.7274796Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_math_ops/TEST-DistMathOpsTest-20230111211958.xml
2023-01-11T21:20:04.7275046Z 
2023-01-11T21:20:04.7275369Z ##[endgroup]
2023-01-11T21:20:04.7275797Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_math_ops_v8hr1mbo)
2023-01-11T21:20:04.7276049Z 
2023-01-11T21:20:04.7276262Z Running distributed/test_c10d_object_collectives ... [2023-01-11 21:20:04.723190]
2023-01-11T21:20:04.7276802Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:20:04.723639]
2023-01-11T21:20:13.0425122Z 
2023-01-11T21:20:13.0425775Z Expand the folded group to see the log file of distributed/test_c10d_object_collectives
2023-01-11T21:20:13.0426939Z ##[group]PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_object_collectives_542zgxbi)
2023-01-11T21:20:13.0427687Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_object_collectives
2023-01-11T21:20:13.0428002Z 
2023-01-11T21:20:13.0428139Z Running tests...
2023-01-11T21:20:13.0428704Z ----------------------------------------------------------------------
2023-01-11T21:20:13.0429215Z   test_all_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26899
2023-01-11T21:20:13.0429656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26900
2023-01-11T21:20:13.0430181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:13.0430524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:13.0431035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:13.0431377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:13.0431755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:13.0432224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:13.0432719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:13.0433699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:13.0434472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:13.0435060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:13.0435675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:13.0436278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:13.0436861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:20:13.0437393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:20:13.0437674Z ok (1.878s)
2023-01-11T21:20:13.0438012Z   test_broadcast_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26927
2023-01-11T21:20:13.0438420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26928
2023-01-11T21:20:13.0438864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:13.0439204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:13.0439639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:13.0439993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:13.0440314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:13.0440772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:13.0441103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:13.0441603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:13.0441942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:13.0442273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:13.0442642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:13.0443007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:13.0443511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:20:13.0444034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:20:13.0444324Z ok (1.613s)
2023-01-11T21:20:13.0444629Z   test_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26955
2023-01-11T21:20:13.0445019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26956
2023-01-11T21:20:13.0445474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:13.0445793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:13.0446228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:13.0446579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:13.0446904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:13.0447351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:13.0447687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:13.0448115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:13.0448461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:13.0448770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:13.0449129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:13.0449546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:13.0450096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:20:13.0450626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:20:13.0450918Z ok (1.612s)
2023-01-11T21:20:13.0451247Z   test_scatter_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26983
2023-01-11T21:20:13.0451630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26984
2023-01-11T21:20:13.0452086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:13.0452421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:13.0452857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:13.0453196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:13.0453524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:13.0453983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:13.0454303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:13.0454736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:13.0455083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:13.0455436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:13.0455792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:13.0456163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:13.0456657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:20:13.0457180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:20:13.0457458Z ok (1.612s)
2023-01-11T21:20:13.0457563Z 
2023-01-11T21:20:13.0457765Z ----------------------------------------------------------------------
2023-01-11T21:20:13.0458013Z Ran 4 tests in 6.716s
2023-01-11T21:20:13.0458131Z 
2023-01-11T21:20:13.0458183Z OK
2023-01-11T21:20:13.0458277Z 
2023-01-11T21:20:13.0458366Z Generating XML reports...
2023-01-11T21:20:13.0458840Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20230111212005.xml
2023-01-11T21:20:13.0459115Z 
2023-01-11T21:20:13.0459400Z ##[endgroup]
2023-01-11T21:20:13.0459872Z FINISHED PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_object_collectives_542zgxbi)
2023-01-11T21:20:13.0460147Z 
2023-01-11T21:20:13.0460357Z Running distributed/_composable/test_replicate ... [2023-01-11 21:20:13.043197]
2023-01-11T21:20:13.0460904Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_replicate.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:20:13.043755]
2023-01-11T21:20:24.0721747Z 
2023-01-11T21:20:24.0722917Z Expand the folded group to see the log file of distributed/_composable/test_replicate
2023-01-11T21:20:24.0724121Z ##[group]PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_replicate_qzw3dwhf)
2023-01-11T21:20:24.0725588Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_replicate
2023-01-11T21:20:24.0726041Z 
2023-01-11T21:20:24.0726336Z Running tests...
2023-01-11T21:20:24.0728194Z ----------------------------------------------------------------------
2023-01-11T21:20:24.0728808Z   test_replicate_non_root_multiple_save_load (__main__.ReplicateStateDictTest)
2023-01-11T21:20:24.0729457Z Tests tha replicate() on multiple submodules matches ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27022
2023-01-11T21:20:24.0730104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27023
2023-01-11T21:20:24.0730648Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27024
2023-01-11T21:20:24.0731296Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27025
2023-01-11T21:20:24.0732025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0732576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0733308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0733824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0734279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:24.0734929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0735385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0736096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0736690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0737235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:24.0738062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0738572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0739202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0739684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0740134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:24.0740768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0741201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0742023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0742489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0742924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:24.0743261Z ok (2.077s)
2023-01-11T21:20:24.0743577Z   test_replicate_single_module_save_load (__main__.ReplicateStateDictTest)
2023-01-11T21:20:24.0744097Z Tests that replicate() on a single module state_dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27066
2023-01-11T21:20:24.0744615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27067
2023-01-11T21:20:24.0745058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27068
2023-01-11T21:20:24.0745487Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27069
2023-01-11T21:20:24.0746123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0746740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0747402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0747866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0748302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:24.0748926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0749341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0749926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0750381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0750805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:24.0751413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0751856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0752427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0752892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0753306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:24.0753937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0754409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0755017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0755513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0755961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:24.0756321Z ok (1.719s)
2023-01-11T21:20:24.0756770Z   test_replicate_multi_module (__main__.ReplicateTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27110
2023-01-11T21:20:24.0757314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27111
2023-01-11T21:20:24.0757808Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27112
2023-01-11T21:20:24.0758276Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27113
2023-01-11T21:20:24.0758922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0759365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0759946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0760391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0760813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:24.0761514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0761938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0762524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0762989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0763472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:24.0764169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0764765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0765496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0765987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0766416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:24.0767076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0767538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0768120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0768596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0769063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:24.0769571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:24.0770118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:24.0770718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:24.0771303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:24.0772126Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:24.0772955Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:24.0773744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:24.0774496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:24.0775021Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0775601Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0776211Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0776822Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0777421Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0777997Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0778618Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0779180Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0779697Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0780371Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0780861Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0781307Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0781643Z ok (1.820s)
2023-01-11T21:20:24.0782210Z   test_replicate_single_module (__main__.ReplicateTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27166
2023-01-11T21:20:24.0782734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27167
2023-01-11T21:20:24.0783319Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27168
2023-01-11T21:20:24.0783756Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27169
2023-01-11T21:20:24.0784493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0784923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0785522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0786032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0786557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:24.0787280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0787811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0788401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0788860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0789293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:24.0789916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0790350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0790916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0791374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0791800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:24.0792431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0792858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0793447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0793908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0794321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:24.0794829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:24.0795365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:24.0795895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:24.0796657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:24.0797288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:24.0798077Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:24.0798849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:24.0799611Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:24.0800181Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0800651Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0801100Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0801727Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0802061Z ok (1.820s)
2023-01-11T21:20:24.0802531Z   test_replicate_with_kwargs (__main__.ReplicateTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27222
2023-01-11T21:20:24.0803035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27223
2023-01-11T21:20:24.0803478Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27224
2023-01-11T21:20:24.0803914Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27225
2023-01-11T21:20:24.0804578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0805104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0805772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0806326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0806825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:24.0807581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0808049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0808654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0809121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0809571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:24.0810207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0810669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0811317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0811824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0812309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:24.0813029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:24.0813524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:24.0814217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:24.0814749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:24.0815297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:24.0815946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:24.0816614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:24.0817253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:24.0817900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:24.0818724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:24.0819612Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:24.0820479Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:24.0821494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:24.0822389Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0823043Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0823665Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0824248Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:20:24.0824651Z ok (1.921s)
2023-01-11T21:20:24.0824835Z 
2023-01-11T21:20:24.0825191Z ----------------------------------------------------------------------
2023-01-11T21:20:24.0825581Z Ran 5 tests in 9.358s
2023-01-11T21:20:24.0825778Z 
2023-01-11T21:20:24.0825881Z OK
2023-01-11T21:20:24.0826033Z 
2023-01-11T21:20:24.0826166Z Generating XML reports...
2023-01-11T21:20:24.0827006Z Generated XML report: test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateStateDictTest-20230111212014.xml
2023-01-11T21:20:24.0828106Z Generated XML report: test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateTest-20230111212014.xml
2023-01-11T21:20:24.0828557Z 
2023-01-11T21:20:24.0829201Z ##[endgroup]
2023-01-11T21:20:24.0830044Z FINISHED PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_replicate_qzw3dwhf)
2023-01-11T21:20:24.0830533Z 
2023-01-11T21:20:24.0830914Z Running distributed/fsdp/test_fsdp_tp_integration ... [2023-01-11 21:20:24.072678]
2023-01-11T21:20:24.0831919Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:20:24.073135]
2023-01-11T21:20:35.4191096Z 
2023-01-11T21:20:35.4191655Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_tp_integration
2023-01-11T21:20:35.4192818Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_vq7m88o3)
2023-01-11T21:20:35.4193637Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration
2023-01-11T21:20:35.4193851Z 
2023-01-11T21:20:35.4193924Z Running tests...
2023-01-11T21:20:35.4194232Z ----------------------------------------------------------------------
2023-01-11T21:20:35.4194672Z   test_fsdp_tp_checkpoint_integration (__main__.TestTPFSDPIntegration)
2023-01-11T21:20:35.4195257Z Tests checkpointing for TP + FSDP integration. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27289
2023-01-11T21:20:35.4195870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27290
2023-01-11T21:20:35.4196330Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27291
2023-01-11T21:20:35.4196658Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27292
2023-01-11T21:20:35.4197141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4197478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4197901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4198248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4198585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:35.4199058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4199375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4199802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4200467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4200989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:35.4201637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4201971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4202400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4202732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4203066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:35.4203588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4203967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4204439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4204805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4205172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:35.4205671Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4206184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4206700Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4207219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4207590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:35.4207938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:35.4208276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:35.4208622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:35.4208867Z dist init r=1, world=4
2023-01-11T21:20:35.4209046Z dist init r=3, world=4
2023-01-11T21:20:35.4209223Z dist init r=0, world=4
2023-01-11T21:20:35.4209384Z dist init r=2, world=4
2023-01-11T21:20:35.4209588Z skip: Need at least 4 CUDA devices (2.179s)
2023-01-11T21:20:35.4209932Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2023-01-11T21:20:35.4210501Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27345
2023-01-11T21:20:35.4210889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27346
2023-01-11T21:20:35.4211215Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27347
2023-01-11T21:20:35.4211542Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27348
2023-01-11T21:20:35.4211982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4212310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4212736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4213080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4213469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:35.4213972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4214301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4214717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4215060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4215393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:35.4215860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4216173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4216598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4216942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4217277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:35.4217761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4218259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4218585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4219002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4219345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4219677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:35.4220170Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4220678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4221192Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4221578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:35.4222180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:35.4222740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:35.4223338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:35.4223825Z dist init r=0, world=4
2023-01-11T21:20:35.4224016Z dist init r=1, world=4
2023-01-11T21:20:35.4224195Z dist init r=3, world=4
2023-01-11T21:20:35.4224372Z dist init r=2, world=4
2023-01-11T21:20:35.4224566Z skip: Need at least 4 CUDA devices (1.821s)
2023-01-11T21:20:35.4224907Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2023-01-11T21:20:35.4225483Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27401
2023-01-11T21:20:35.4225880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27402
2023-01-11T21:20:35.4226196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27403
2023-01-11T21:20:35.4226516Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27404
2023-01-11T21:20:35.4226970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4227398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4227858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4228210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4228548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:35.4229004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4229333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4229762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4230103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4230424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:35.4230894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4231222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4231635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4231974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4232308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:35.4232771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4233083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4233507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4233853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4234186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:35.4234662Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4235178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4235695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4236207Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4236577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:35.4236926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:35.4237270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:35.4237601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:35.4237853Z dist init r=3, world=4
2023-01-11T21:20:35.4238035Z dist init r=2, world=4
2023-01-11T21:20:35.4238199Z dist init r=0, world=4
2023-01-11T21:20:35.4238375Z dist init r=1, world=4
2023-01-11T21:20:35.4238578Z skip: Need at least 4 CUDA devices (1.825s)
2023-01-11T21:20:35.4238922Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2023-01-11T21:20:35.4239473Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27457
2023-01-11T21:20:35.4239909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27458
2023-01-11T21:20:35.4240237Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27459
2023-01-11T21:20:35.4240578Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27460
2023-01-11T21:20:35.4241028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4241360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4241863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4242195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4242526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:35.4242997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4243324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4243740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4244079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4244410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:35.4244891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4245386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4245711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4246137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4246467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4246798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:35.4247266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4247577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4247999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4248340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4248673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:35.4249156Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4249673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4250190Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4250569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:35.4250902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:35.4251238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:35.4251578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:35.4251824Z dist init r=0, world=4
2023-01-11T21:20:35.4265684Z dist init r=3, world=4
2023-01-11T21:20:35.4266087Z dist init r=2, world=4
2023-01-11T21:20:35.4266325Z dist init r=1, world=4
2023-01-11T21:20:35.4266796Z skip: Need at least 4 CUDA devices (1.822s)
2023-01-11T21:20:35.4267157Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2023-01-11T21:20:35.4267839Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27513
2023-01-11T21:20:35.4268247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27514
2023-01-11T21:20:35.4268582Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27515
2023-01-11T21:20:35.4268916Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27516
2023-01-11T21:20:35.4269391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4269728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4270161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4270514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4270842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:35.4271316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4271647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4272074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4272402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4272738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:35.4273206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4273524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4273956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4274302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4274635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:35.4275092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:35.4275420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:35.4275848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:35.4276190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:35.4276512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:35.4277012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4277536Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4278040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4278556Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:35.4278943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:35.4279291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:35.4279625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:35.4279998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:35.4280257Z dist init r=0, world=4
2023-01-11T21:20:35.4280459Z dist init r=2, world=4
2023-01-11T21:20:35.4280639Z dist init r=3, world=4
2023-01-11T21:20:35.4280815Z dist init r=1, world=4
2023-01-11T21:20:35.4281006Z skip: Need at least 4 CUDA devices (2.023s)
2023-01-11T21:20:35.4281144Z 
2023-01-11T21:20:35.4281347Z ----------------------------------------------------------------------
2023-01-11T21:20:35.4281671Z Ran 5 tests in 9.671s
2023-01-11T21:20:35.4281787Z 
2023-01-11T21:20:35.4281861Z OK (skipped=5)
2023-01-11T21:20:35.4281967Z 
2023-01-11T21:20:35.4282039Z Generating XML reports...
2023-01-11T21:20:35.4282524Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20230111212025.xml
2023-01-11T21:20:35.4282805Z 
2023-01-11T21:20:35.4283116Z ##[endgroup]
2023-01-11T21:20:35.4283590Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_vq7m88o3)
2023-01-11T21:20:35.4283870Z 
2023-01-11T21:20:35.4284079Z Running distributed/_tensor/test_tp_sharding_ops ... [2023-01-11 21:20:35.419966]
2023-01-11T21:20:35.4284615Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tp_sharding_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:20:35.420318]
2023-01-11T21:20:47.9635263Z 
2023-01-11T21:20:47.9636119Z Expand the folded group to see the log file of distributed/_tensor/test_tp_sharding_ops
2023-01-11T21:20:47.9637326Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tp_sharding_ops_tws96xfq)
2023-01-11T21:20:47.9638452Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops
2023-01-11T21:20:47.9638787Z 
2023-01-11T21:20:47.9638866Z Running tests...
2023-01-11T21:20:47.9639180Z ----------------------------------------------------------------------
2023-01-11T21:20:47.9639591Z   test_replicated_permute (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27580
2023-01-11T21:20:47.9640218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27581
2023-01-11T21:20:47.9644050Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27582
2023-01-11T21:20:47.9644736Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27583
2023-01-11T21:20:47.9645711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9646306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9646958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9647323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9647662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:47.9648137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9651192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9651961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9652331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9652664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:47.9653124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9653679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9654195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9654552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9671158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:47.9672124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9672768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9673575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9674212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9674657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:47.9675262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:47.9675891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:47.9676584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:47.9677278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:47.9677937Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9678492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9679027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9679553Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9679870Z ok (2.080s)
2023-01-11T21:20:47.9680215Z   test_sharded_cat (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27636
2023-01-11T21:20:47.9680611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27637
2023-01-11T21:20:47.9680942Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27638
2023-01-11T21:20:47.9681279Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27639
2023-01-11T21:20:47.9681843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9682205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9683161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9683617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9683949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:47.9684445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9684787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9685229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9685581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9685913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:47.9686389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9686924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9687351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9687771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9688096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:47.9688563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9688902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9689337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9689688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9690000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:47.9690372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:47.9690744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:47.9691117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:47.9691469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:47.9691972Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9692499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9698710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9699528Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9700007Z ok (1.720s)
2023-01-11T21:20:47.9700488Z   test_sharded_permute (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27692
2023-01-11T21:20:47.9701060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27693
2023-01-11T21:20:47.9701510Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27694
2023-01-11T21:20:47.9702182Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27695
2023-01-11T21:20:47.9702835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9703281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9703906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9704402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9704875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:47.9705574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9706109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9706793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9707331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9707850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:47.9708584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9709263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9709864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9710461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9710972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:47.9711769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9712348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9713144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9713758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9714327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:47.9714984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:47.9715645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:47.9716304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:47.9716965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:47.9717896Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9718796Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9719583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9720366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9720793Z ok (1.720s)
2023-01-11T21:20:47.9721242Z   test_sharded_split (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27748
2023-01-11T21:20:47.9721918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27749
2023-01-11T21:20:47.9722426Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27750
2023-01-11T21:20:47.9722902Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27751
2023-01-11T21:20:47.9723593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9724089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9724791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9725358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9725901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:47.9726540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9726990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9727645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9728197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9728710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:47.9729385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9729721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9730274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9730672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9731004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:47.9731462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9731796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9732233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9732619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9732949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:47.9733306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:47.9733678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:47.9734048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:47.9734541Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9735063Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9735464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:47.9735960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9736462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9736752Z ok (1.819s)
2023-01-11T21:20:47.9737077Z   test_sharded_transpose (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27804
2023-01-11T21:20:47.9737469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27805
2023-01-11T21:20:47.9737787Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27806
2023-01-11T21:20:47.9738115Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27807
2023-01-11T21:20:47.9738574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9738897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9739339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9739697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9740026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:47.9740486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9740815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9741250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9741603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9742139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:47.9742619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9743120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9743896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9744575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9745188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:47.9746075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9746619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9747455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9747828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9748142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:47.9748515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:47.9748890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:47.9749262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:47.9749613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:47.9750116Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9751058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9751703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9752208Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9752501Z ok (1.720s)
2023-01-11T21:20:47.9752835Z   test_sharded_view (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27860
2023-01-11T21:20:47.9753231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27861
2023-01-11T21:20:47.9753555Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27862
2023-01-11T21:20:47.9753880Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27863
2023-01-11T21:20:47.9754354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9754689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9755113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9755473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9755798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:20:47.9756252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9756585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9757015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9757364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9757676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:20:47.9758134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9758466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9758957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9759344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9759673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:20:47.9760140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:20:47.9760459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:20:47.9760892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:20:47.9761236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:20:47.9761649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:20:47.9762008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:20:47.9762381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:20:47.9762750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:20:47.9763236Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9763634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:20:47.9764124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9764641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9765146Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:20:47.9765442Z ok (1.819s)
2023-01-11T21:20:47.9765554Z 
2023-01-11T21:20:47.9765770Z ----------------------------------------------------------------------
2023-01-11T21:20:47.9766019Z Ran 6 tests in 10.879s
2023-01-11T21:20:47.9766123Z 
2023-01-11T21:20:47.9766188Z OK
2023-01-11T21:20:47.9766283Z 
2023-01-11T21:20:47.9766371Z Generating XML reports...
2023-01-11T21:20:47.9766842Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops/TEST-TPShardingOpsTest-20230111212036.xml
2023-01-11T21:20:47.9767109Z 
2023-01-11T21:20:47.9767512Z ##[endgroup]
2023-01-11T21:20:47.9767991Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tp_sharding_ops_tws96xfq)
2023-01-11T21:20:47.9768260Z 
2023-01-11T21:20:47.9768484Z Running distributed/tensor/parallel/test_parallelize_api ... [2023-01-11 21:20:47.963901]
2023-01-11T21:20:47.9769050Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/tensor/parallel/test_parallelize_api.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:20:47.964287]
2023-01-11T21:21:00.9181359Z 
2023-01-11T21:21:00.9182494Z Expand the folded group to see the log file of distributed/tensor/parallel/test_parallelize_api
2023-01-11T21:21:00.9183713Z ##[group]PRINTING LOG FILE of distributed/tensor/parallel/test_parallelize_api (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_parallelize_api_elkfoau2)
2023-01-11T21:21:00.9184812Z Test results will be stored in test-reports/python-unittest/distributed.tensor.parallel.test_parallelize_api
2023-01-11T21:21:00.9185367Z 
2023-01-11T21:21:00.9185501Z Running tests...
2023-01-11T21:21:00.9186058Z ----------------------------------------------------------------------
2023-01-11T21:21:00.9186770Z   test_creat_1d_device_mesh (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27927
2023-01-11T21:21:00.9187809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27928
2023-01-11T21:21:00.9188536Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27929
2023-01-11T21:21:00.9189152Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27930
2023-01-11T21:21:00.9190013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9190629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9191428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9192046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9192589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:00.9193368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9193928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9194736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9195367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9195962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:00.9196811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9197407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9198210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9198572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9198885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:00.9199349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9199676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9200108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9200437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9200761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:00.9201123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:00.9201577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:00.9201947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:21:00.9202321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:21:00.9202818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9203334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9203857Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9204371Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9204766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:21:00.9205216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T21:21:00.9205617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:21:00.9205982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:21:00.9206475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:21:00.9206976Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:21:00.9207359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:21:00.9207717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T21:21:00.9208194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:21:00.9208707Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:21:00.9209100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:21:00.9209460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T21:21:00.9209935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:21:00.9210445Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:21:00.9210835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T21:21:00.9211318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:21:00.9211824Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:21:00.9212222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T21:21:00.9212582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:21:00.9212939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:21:00.9213526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:21:00.9214036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:21:00.9214431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T21:21:00.9214791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T21:21:00.9215271Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:21:00.9215783Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:21:00.9216174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T21:21:00.9216523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T21:21:00.9217002Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:21:00.9217513Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:21:00.9218020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:21:00.9218601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:21:00.9218887Z ok (2.191s)
2023-01-11T21:21:00.9219228Z   test_creat_1d_device_mesh_error (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28007
2023-01-11T21:21:00.9219640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28008
2023-01-11T21:21:00.9219963Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28009
2023-01-11T21:21:00.9220285Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28010
2023-01-11T21:21:00.9220737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9221055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9221485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9222116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9222446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:00.9222901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9223233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9223666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9224019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9224330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:00.9224793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9225123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9225537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9225882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9226207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:00.9226668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9226982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9227410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9227754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9228065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:00.9228434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:00.9228799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:21:00.9229162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:00.9229642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9230157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9230546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:21:00.9231111Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9231661Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9231946Z ok (1.819s)
2023-01-11T21:21:00.9232279Z   test_linear_col_wise_parallel (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28063
2023-01-11T21:21:00.9232678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28064
2023-01-11T21:21:00.9232995Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28065
2023-01-11T21:21:00.9233319Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28066
2023-01-11T21:21:00.9233772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9234094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9234529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9234883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9235208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:00.9235653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9235979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9236405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9236736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9237055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:00.9237516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9237846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9238264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9238603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9238926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:00.9239385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9239699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9240128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9240477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9240783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:00.9241146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:00.9241611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:00.9241974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:21:00.9242322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:21:00.9242818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9243334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9243899Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9244437Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9244736Z ok (1.822s)
2023-01-11T21:21:00.9245072Z   test_linear_row_wise_parallel (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28119
2023-01-11T21:21:00.9245469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28120
2023-01-11T21:21:00.9245805Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28121
2023-01-11T21:21:00.9246128Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28122
2023-01-11T21:21:00.9246581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9246897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9247328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9247675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9247983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:00.9248446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9248773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9249202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9249531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9249852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:00.9250314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9250641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9251058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9251404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9251726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:00.9252173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9252506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9252931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9253277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9253584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:00.9253946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:00.9254311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:21:00.9254659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:00.9255017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:21:00.9255507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9256033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9256586Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9257131Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9257418Z ok (1.819s)
2023-01-11T21:21:00.9257745Z   test_parallelize_mlp (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28175
2023-01-11T21:21:00.9258129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28176
2023-01-11T21:21:00.9258458Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28177
2023-01-11T21:21:00.9258780Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28178
2023-01-11T21:21:00.9259223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9259559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9259991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9260343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9260652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:00.9261109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9261435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9261975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9262310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9262632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:00.9263096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9263413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9263842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9264189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9264513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:00.9264959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9265284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9265710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9266043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9266361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:00.9266725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:00.9267093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:00.9267441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:21:00.9267802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:21:00.9268289Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9268810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9269395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9269972Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9270263Z ok (1.820s)
2023-01-11T21:21:00.9270584Z   test_parallelize_mlp_error (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28231
2023-01-11T21:21:00.9270982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28232
2023-01-11T21:21:00.9271311Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28233
2023-01-11T21:21:00.9271638Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28234
2023-01-11T21:21:00.9272083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9272418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9272856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9273207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9273522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:00.9273978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9274305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9274719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9275065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9275381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:00.9275840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9276156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9276582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9276925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9277236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:00.9277694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:00.9278023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:00.9278447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:00.9278778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:00.9279098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:00.9279460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:00.9279825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:00.9280172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:21:00.9280532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:21:00.9281020Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9281606Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9282180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9282758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:00.9283039Z ok (1.820s)
2023-01-11T21:21:00.9283142Z 
2023-01-11T21:21:00.9283332Z ----------------------------------------------------------------------
2023-01-11T21:21:00.9283580Z Ran 6 tests in 11.292s
2023-01-11T21:21:00.9283697Z 
2023-01-11T21:21:00.9283759Z OK
2023-01-11T21:21:00.9283852Z 
2023-01-11T21:21:00.9283940Z Generating XML reports...
2023-01-11T21:21:00.9284439Z Generated XML report: test-reports/python-unittest/distributed.tensor.parallel.test_parallelize_api/TEST-TensorParallelAPITests-20230111212049.xml
2023-01-11T21:21:00.9284740Z 
2023-01-11T21:21:00.9285084Z ##[endgroup]
2023-01-11T21:21:00.9285605Z FINISHED PRINTING LOG FILE of distributed/tensor/parallel/test_parallelize_api (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_parallelize_api_elkfoau2)
2023-01-11T21:21:00.9285914Z 
2023-01-11T21:21:00.9286090Z Running distributed/test_c10d_common ... [2023-01-11 21:21:00.919089]
2023-01-11T21:21:00.9286616Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:21:00.919605]
2023-01-11T21:21:36.2361176Z 
2023-01-11T21:21:36.2362036Z Expand the folded group to see the log file of distributed/test_c10d_common
2023-01-11T21:21:36.2363061Z ##[group]PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_hor6egn8)
2023-01-11T21:21:36.2363698Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_debug_level>]>
2023-01-11T21:21:36.2364108Z test_debug_level (__main__.CommTest)
2023-01-11T21:21:36.2365052Z <unittest.suite.TestSuite tests=[<__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_single_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_single_dtype>]>
2023-01-11T21:21:36.2366113Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2023-01-11T21:21:36.2366572Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2023-01-11T21:21:36.2367017Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2023-01-11T21:21:36.2367335Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2023-01-11T21:21:36.2367994Z <unittest.suite.TestSuite tests=[<__main__.PythonProcessGroupExtensionTest testMethod=test_backend_class_attr>, <__main__.PythonProcessGroupExtensionTest testMethod=test_collectives>, <__main__.PythonProcessGroupExtensionTest testMethod=test_get_backend_name>, <__main__.PythonProcessGroupExtensionTest testMethod=test_send_recv>]>
2023-01-11T21:21:36.2368629Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest)
2023-01-11T21:21:36.2368938Z test_collectives (__main__.PythonProcessGroupExtensionTest)
2023-01-11T21:21:36.2369306Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest)
2023-01-11T21:21:36.2369620Z test_send_recv (__main__.PythonProcessGroupExtensionTest)
2023-01-11T21:21:36.2370228Z <unittest.suite.TestSuite tests=[<__main__.ReduceOpTest testMethod=test_op_isinstance_of_reduceop>, <__main__.ReduceOpTest testMethod=test_reduceop_copyable>, <__main__.ReduceOpTest testMethod=test_reduceop_equal>, <__main__.ReduceOpTest testMethod=test_reduceop_pickle>]>
2023-01-11T21:21:36.2370749Z test_op_isinstance_of_reduceop (__main__.ReduceOpTest)
2023-01-11T21:21:36.2371005Z test_reduceop_copyable (__main__.ReduceOpTest)
2023-01-11T21:21:36.2371246Z test_reduceop_equal (__main__.ReduceOpTest)
2023-01-11T21:21:36.2371473Z test_reduceop_pickle (__main__.ReduceOpTest)
2023-01-11T21:21:36.2372161Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T21:21:36.2372737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2373149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2373666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2374017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2374185Z 
2023-01-11T21:21:36.2374261Z Running tests...
2023-01-11T21:21:36.2374558Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2374926Z   test_debug_level (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28308
2023-01-11T21:21:36.2375295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28309
2023-01-11T21:21:36.2375756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2376081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2376514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2376860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2377171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:36.2377634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2377967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2378396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2378732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2379057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:36.2379314Z ok (1.869s)
2023-01-11T21:21:36.2379415Z 
2023-01-11T21:21:36.2379604Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2379849Z Ran 1 test in 1.869s
2023-01-11T21:21:36.2379962Z 
2023-01-11T21:21:36.2380026Z OK
2023-01-11T21:21:36.2380117Z 
2023-01-11T21:21:36.2380204Z Generating XML reports...
2023-01-11T21:21:36.2380605Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20230111212103.xml
2023-01-11T21:21:36.2381066Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T21:21:36.2381529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2382127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2382571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2382921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2383086Z 
2023-01-11T21:21:36.2383161Z Running tests...
2023-01-11T21:21:36.2383456Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2383785Z   test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.259s)
2023-01-11T21:21:36.2383974Z 
2023-01-11T21:21:36.2384172Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2384396Z Ran 1 test in 0.259s
2023-01-11T21:21:36.2384509Z 
2023-01-11T21:21:36.2384571Z OK
2023-01-11T21:21:36.2384662Z 
2023-01-11T21:21:36.2384750Z Generating XML reports...
2023-01-11T21:21:36.2385223Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111212106.xml
2023-01-11T21:21:36.2385812Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T21:21:36.2386325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2386662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2387094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2387428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2387593Z 
2023-01-11T21:21:36.2387666Z Running tests...
2023-01-11T21:21:36.2387969Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2388285Z   test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.260s)
2023-01-11T21:21:36.2388475Z 
2023-01-11T21:21:36.2388672Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2388914Z Ran 1 test in 0.261s
2023-01-11T21:21:36.2389026Z 
2023-01-11T21:21:36.2389088Z OK
2023-01-11T21:21:36.2389165Z 
2023-01-11T21:21:36.2389255Z Generating XML reports...
2023-01-11T21:21:36.2389727Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111212108.xml
2023-01-11T21:21:36.2390230Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T21:21:36.2390683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2391013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2391446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2391793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2391945Z 
2023-01-11T21:21:36.2392019Z Running tests...
2023-01-11T21:21:36.2392319Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2392647Z   test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.256s)
2023-01-11T21:21:36.2392833Z 
2023-01-11T21:21:36.2393015Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2393257Z Ran 1 test in 0.256s
2023-01-11T21:21:36.2393369Z 
2023-01-11T21:21:36.2393433Z OK
2023-01-11T21:21:36.2393524Z 
2023-01-11T21:21:36.2393607Z Generating XML reports...
2023-01-11T21:21:36.2394062Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111212110.xml
2023-01-11T21:21:36.2394566Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T21:21:36.2395033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2395354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2395786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2396134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2396296Z 
2023-01-11T21:21:36.2396371Z Running tests...
2023-01-11T21:21:36.2396663Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2396991Z   test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.266s)
2023-01-11T21:21:36.2397181Z 
2023-01-11T21:21:36.2397380Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2397605Z Ran 1 test in 0.266s
2023-01-11T21:21:36.2397720Z 
2023-01-11T21:21:36.2397785Z OK
2023-01-11T21:21:36.2397879Z 
2023-01-11T21:21:36.2397964Z Generating XML reports...
2023-01-11T21:21:36.2398437Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111212112.xml
2023-01-11T21:21:36.2398976Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T21:21:36.2399476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2399818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2400233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2400581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2400745Z 
2023-01-11T21:21:36.2400820Z Running tests...
2023-01-11T21:21:36.2401127Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2401621Z   test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28381
2023-01-11T21:21:36.2402044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28382
2023-01-11T21:21:36.2402378Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28383
2023-01-11T21:21:36.2402708Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28384
2023-01-11T21:21:36.2403156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2403489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2403919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2404251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2404580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:36.2405043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2405378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2405799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2406144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2406466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:36.2406912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2407243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2407669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2408013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2408325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:36.2408789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2409117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2409545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2409882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2410210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:36.2410467Z ok (1.984s)
2023-01-11T21:21:36.2410571Z 
2023-01-11T21:21:36.2410758Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2410999Z Ran 1 test in 1.984s
2023-01-11T21:21:36.2411112Z 
2023-01-11T21:21:36.2411177Z OK
2023-01-11T21:21:36.2411316Z 
2023-01-11T21:21:36.2411401Z Generating XML reports...
2023-01-11T21:21:36.2411908Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111212114.xml
2023-01-11T21:21:36.2412430Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T21:21:36.2412891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2413211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2413641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2413988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2414151Z 
2023-01-11T21:21:36.2414224Z Running tests...
2023-01-11T21:21:36.2414514Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2414931Z   test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28436
2023-01-11T21:21:36.2415345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28437
2023-01-11T21:21:36.2415662Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28438
2023-01-11T21:21:36.2415989Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28439
2023-01-11T21:21:36.2416440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2416771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2417187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2417538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2417865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:36.2418316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2418646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2419072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2419421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2419730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:36.2420097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:21:36.2420571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2420903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2421367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2421856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2422264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:36.2422617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:21:36.2423089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2423418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2423847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2424180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2424587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:36.2424994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:36.2425348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:36.2425844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:36.2426369Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:36.2426891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:36.2427388Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:36.2427681Z ok (2.078s)
2023-01-11T21:21:36.2427789Z 
2023-01-11T21:21:36.2427993Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2428238Z Ran 1 test in 2.079s
2023-01-11T21:21:36.2428338Z 
2023-01-11T21:21:36.2428407Z OK
2023-01-11T21:21:36.2428499Z 
2023-01-11T21:21:36.2428587Z Generating XML reports...
2023-01-11T21:21:36.2429084Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111212117.xml
2023-01-11T21:21:36.2429595Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T21:21:36.2430072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2430394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2430825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2431179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2431343Z 
2023-01-11T21:21:36.2431406Z Running tests...
2023-01-11T21:21:36.2431715Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2432135Z   test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28496
2023-01-11T21:21:36.2432552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28497
2023-01-11T21:21:36.2432869Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28498
2023-01-11T21:21:36.2433195Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28499
2023-01-11T21:21:36.2433647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2433962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2434395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2434743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2435069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:36.2435522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2435854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2436286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2436635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2436945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:36.2437413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2437785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2438245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2438595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2438919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:36.2439381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2439698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2440124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2440472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2440783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:36.2441036Z ok (1.976s)
2023-01-11T21:21:36.2441142Z 
2023-01-11T21:21:36.2441426Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2441669Z Ran 1 test in 1.976s
2023-01-11T21:21:36.2441767Z 
2023-01-11T21:21:36.2441833Z OK
2023-01-11T21:21:36.2441925Z 
2023-01-11T21:21:36.2442012Z Generating XML reports...
2023-01-11T21:21:36.2442507Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111212121.xml
2023-01-11T21:21:36.2443014Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T21:21:36.2443483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2443814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2444243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2444576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2444742Z 
2023-01-11T21:21:36.2444814Z Running tests...
2023-01-11T21:21:36.2445114Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2445507Z   test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28551
2023-01-11T21:21:36.2445913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28552
2023-01-11T21:21:36.2446238Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28553
2023-01-11T21:21:36.2446567Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28554
2023-01-11T21:21:36.2447004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2447331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2447760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2448101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2448413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:36.2448889Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2023-01-11T21:21:36.2449371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2449682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2450105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2450498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2450874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:36.2451344Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2023-01-11T21:21:36.2451831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2452163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2452589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2452921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2453239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:36.2453720Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2023-01-11T21:21:36.2454191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2454520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2454949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2455292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2455597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:36.2455960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:36.2456329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:21:36.2456695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:21:36.2457043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:36.2457535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:36.2458055Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:36.2458561Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:36.2459074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:21:36.2459358Z ok (2.983s)
2023-01-11T21:21:36.2459465Z 
2023-01-11T21:21:36.2459664Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2459894Z Ran 1 test in 2.983s
2023-01-11T21:21:36.2460004Z 
2023-01-11T21:21:36.2460067Z OK
2023-01-11T21:21:36.2460157Z 
2023-01-11T21:21:36.2460242Z Generating XML reports...
2023-01-11T21:21:36.2460720Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111212125.xml
2023-01-11T21:21:36.2461233Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T21:21:36.2461873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2462276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2462694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2463044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2463210Z 
2023-01-11T21:21:36.2463361Z Running tests...
2023-01-11T21:21:36.2463664Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2463950Z   test_op_isinstance_of_reduceop (__main__.ReduceOpTest) ... ok (0.269s)
2023-01-11T21:21:36.2464157Z 
2023-01-11T21:21:36.2464362Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2464602Z Ran 1 test in 0.269s
2023-01-11T21:21:36.2464715Z 
2023-01-11T21:21:36.2464765Z OK
2023-01-11T21:21:36.2464857Z 
2023-01-11T21:21:36.2464943Z Generating XML reports...
2023-01-11T21:21:36.2465364Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111212129.xml
2023-01-11T21:21:36.2465829Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T21:21:36.2466276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2466609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2467039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2467376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2467542Z 
2023-01-11T21:21:36.2467619Z Running tests...
2023-01-11T21:21:36.2467921Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2468215Z   test_reduceop_copyable (__main__.ReduceOpTest) ... ok (0.265s)
2023-01-11T21:21:36.2468357Z 
2023-01-11T21:21:36.2468553Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2468785Z Ran 1 test in 0.265s
2023-01-11T21:21:36.2468898Z 
2023-01-11T21:21:36.2468959Z OK
2023-01-11T21:21:36.2469049Z 
2023-01-11T21:21:36.2469121Z Generating XML reports...
2023-01-11T21:21:36.2469537Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111212131.xml
2023-01-11T21:21:36.2470005Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T21:21:36.2470473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2470789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2471212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2471554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2471719Z 
2023-01-11T21:21:36.2471778Z Running tests...
2023-01-11T21:21:36.2472078Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2472364Z   test_reduceop_equal (__main__.ReduceOpTest) ... ok (0.259s)
2023-01-11T21:21:36.2472513Z 
2023-01-11T21:21:36.2472708Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2472936Z Ran 1 test in 0.259s
2023-01-11T21:21:36.2473047Z 
2023-01-11T21:21:36.2473109Z OK
2023-01-11T21:21:36.2473200Z 
2023-01-11T21:21:36.2473283Z Generating XML reports...
2023-01-11T21:21:36.2473695Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111212133.xml
2023-01-11T21:21:36.2474158Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T21:21:36.2474616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:36.2474946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:36.2475359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:36.2475703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:36.2475865Z 
2023-01-11T21:21:36.2475938Z Running tests...
2023-01-11T21:21:36.2476267Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2476560Z   test_reduceop_pickle (__main__.ReduceOpTest) ... ok (0.263s)
2023-01-11T21:21:36.2476712Z 
2023-01-11T21:21:36.2476944Z ----------------------------------------------------------------------
2023-01-11T21:21:36.2477181Z Ran 1 test in 0.264s
2023-01-11T21:21:36.2477281Z 
2023-01-11T21:21:36.2477343Z OK
2023-01-11T21:21:36.2477433Z 
2023-01-11T21:21:36.2477517Z Generating XML reports...
2023-01-11T21:21:36.2477941Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111212135.xml
2023-01-11T21:21:36.2478184Z 
2023-01-11T21:21:36.2478508Z ##[endgroup]
2023-01-11T21:21:36.2478932Z FINISHED PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_hor6egn8)
2023-01-11T21:21:36.2479178Z 
2023-01-11T21:21:36.2479404Z Running distributed/_composable/test_compose ... [2023-01-11 21:21:36.236884]
2023-01-11T21:21:36.2479936Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_compose.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:21:36.237234]
2023-01-11T21:21:51.6888900Z 
2023-01-11T21:21:51.6889556Z Expand the folded group to see the log file of distributed/_composable/test_compose
2023-01-11T21:21:51.6890431Z ##[group]PRINTING LOG FILE of distributed/_composable/test_compose (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_compose_yflxn2d2)
2023-01-11T21:21:51.6891047Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_compose
2023-01-11T21:21:51.6891264Z 
2023-01-11T21:21:51.6891365Z Running tests...
2023-01-11T21:21:51.6891959Z ----------------------------------------------------------------------
2023-01-11T21:21:51.6892648Z   test_checkpoint_fsdp_submodules_non_reentrant (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28651
2023-01-11T21:21:51.6894858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28652
2023-01-11T21:21:51.6895814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6896171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6896615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6896955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6897293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:51.6897769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6898099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6898517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6898873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6940054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:51.6940930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6941958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6942650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:51.6943200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:51.6943450Z dist init r=1, world=2
2023-01-11T21:21:51.6943632Z dist init r=0, world=2
2023-01-11T21:21:51.6943843Z skip: Need at least 2 CUDA devices (1.971s)
2023-01-11T21:21:51.6944389Z   test_checkpoint_fsdp_submodules_use_reentrant (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28679
2023-01-11T21:21:51.6945054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28680
2023-01-11T21:21:51.6945762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6946352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6947151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6947792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6948235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:51.6948696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6949041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6949471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6949816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6950138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:51.6950637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6951160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6951546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:51.6951880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:51.6952135Z dist init r=0, world=2
2023-01-11T21:21:51.6952315Z dist init r=1, world=2
2023-01-11T21:21:51.6952509Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T21:21:51.6952880Z   test_checkpoint_fsdp_submodules_with_param (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28707
2023-01-11T21:21:51.6953288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28708
2023-01-11T21:21:51.6953741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6954059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6954488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6954832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6955157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:51.6955629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6955973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6956400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6956728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6957061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:51.6957554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6958073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6958527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:51.6958914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:51.6959174Z dist init r=0, world=2
2023-01-11T21:21:51.6959342Z dist init r=1, world=2
2023-01-11T21:21:51.6959548Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:21:51.6959936Z   test_checkpoint_fsdp_submodules_with_param_no_shard (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28735
2023-01-11T21:21:51.6960356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28736
2023-01-11T21:21:51.6960796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6961123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6961635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6961986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6962312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:51.6962785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6963111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6963526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6963870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6964201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:51.6964696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6965213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6970011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:51.6970582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:51.6970993Z dist init r=0, world=2
2023-01-11T21:21:51.6971177Z dist init r=1, world=2
2023-01-11T21:21:51.6971409Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:21:51.6972028Z   test_composable_fsdp_replicate (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28763
2023-01-11T21:21:51.6972667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28764
2023-01-11T21:21:51.6973533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6974141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6974946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6975579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6976188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:51.6976917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6977468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6978272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6978893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6979373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:51.6979930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6980456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6980832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:51.6981181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:51.6981441Z dist init r=1, world=2
2023-01-11T21:21:51.6981610Z dist init r=0, world=2
2023-01-11T21:21:51.6982010Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:21:51.6982288Z   test_fully_shard_replicate_composability (__main__.TestFSDPCheckpoint)
2023-01-11T21:21:51.6982684Z Tests composing ``fully_shard`` and ``replicate``. To save unit test ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28791
2023-01-11T21:21:51.6983070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28792
2023-01-11T21:21:51.6983542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6983876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6984293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6984640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6984978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:51.6985449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6985764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6986195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6986545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6986881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:51.6987402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6987924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6988313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:51.6988660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:51.6988907Z dist init r=0, world=2
2023-01-11T21:21:51.6989092Z dist init r=1, world=2
2023-01-11T21:21:51.6989300Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:21:51.6989665Z   test_wrap_same_submodule_use_reentrant_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28819
2023-01-11T21:21:51.6990075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28820
2023-01-11T21:21:51.6990527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6990858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6991274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6991619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6991955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:51.6992517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6992873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6993302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6993643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6993963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:51.6994456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6994972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.6995357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:51.6995692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:51.6995945Z dist init r=1, world=2
2023-01-11T21:21:51.6996134Z dist init r=0, world=2
2023-01-11T21:21:51.6996328Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:21:51.6996698Z   test_wrap_same_submodule_use_reentrant_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28847
2023-01-11T21:21:51.6997103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28848
2023-01-11T21:21:51.6997556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6997873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.6998297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.6998648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.6998971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:21:51.6999440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:51.6999768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:51.7000193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:51.7000525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:51.7000859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:21:51.7001422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.7001945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:21:51.7002322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:51.7002672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:51.7002931Z dist init r=1, world=2
2023-01-11T21:21:51.7003098Z dist init r=0, world=2
2023-01-11T21:21:51.7003302Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:21:51.7003439Z 
2023-01-11T21:21:51.7003644Z ----------------------------------------------------------------------
2023-01-11T21:21:51.7003887Z Ran 8 tests in 13.764s
2023-01-11T21:21:51.7003987Z 
2023-01-11T21:21:51.7004057Z OK (skipped=8)
2023-01-11T21:21:51.7004163Z 
2023-01-11T21:21:51.7004247Z Generating XML reports...
2023-01-11T21:21:51.7004701Z Generated XML report: test-reports/python-unittest/distributed._composable.test_compose/TEST-TestFSDPCheckpoint-20230111212137.xml
2023-01-11T21:21:51.7005010Z 
2023-01-11T21:21:51.7005390Z ##[endgroup]
2023-01-11T21:21:51.7005892Z FINISHED PRINTING LOG FILE of distributed/_composable/test_compose (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_compose_yflxn2d2)
2023-01-11T21:21:51.7006159Z 
2023-01-11T21:21:51.7006374Z Running distributed/fsdp/test_fsdp_flatten_params ... [2023-01-11 21:21:51.689402]
2023-01-11T21:21:51.7006912Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_flatten_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:21:51.689874]
2023-01-11T21:22:08.1356816Z 
2023-01-11T21:22:08.1357507Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_flatten_params
2023-01-11T21:22:08.1358576Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_flatten_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_flatten_params_stexuohs)
2023-01-11T21:22:08.1359725Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params
2023-01-11T21:22:08.1360146Z 
2023-01-11T21:22:08.1360277Z Running tests...
2023-01-11T21:22:08.1360836Z ----------------------------------------------------------------------
2023-01-11T21:22:08.1361402Z   test_empty_module (__main__.TestFlattenParams)
2023-01-11T21:22:08.1361944Z Tests flattening an empty module (i.e. one without any parameters). ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28886
2023-01-11T21:22:08.1362718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:08.1363320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:08.1363893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:08.1364246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:08.1364584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:08.1365080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:22:08.1365471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:08.1365736Z dist init r=0, world=1
2023-01-11T21:22:08.1365941Z skip: Need at least 1 CUDA device (1.872s)
2023-01-11T21:22:08.1366187Z   test_flat_param_shard_metadata (__main__.TestFlattenParams)
2023-01-11T21:22:08.1366763Z Tests that ``FlatParameter`` shard metadata are computed as expected. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28900
2023-01-11T21:22:08.1367364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:08.1367686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:08.1368120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:08.1368469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:08.1368809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:08.1369292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:22:08.1369686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:08.1369946Z dist init r=0, world=1
2023-01-11T21:22:08.1370109Z ok (1.611s)
2023-01-11T21:22:08.1370320Z   test_flatten_nothing (__main__.TestFlattenParams)
2023-01-11T21:22:08.1370699Z Tests that constructing a ``FlatParamHandle`` with no parameters ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28914
2023-01-11T21:22:08.1371219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:08.1371732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:08.1372227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:08.1372575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:08.1372915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:08.1373399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:22:08.1373791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:08.1374052Z dist init r=0, world=1
2023-01-11T21:22:08.1374213Z ok (1.608s)
2023-01-11T21:22:08.1374437Z   test_numel_with_shared_params (__main__.TestFlattenParams)
2023-01-11T21:22:08.1374837Z Tests that numel is preserved after flattening when there are shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28928
2023-01-11T21:22:08.1375369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:08.1375689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:08.1376121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:08.1376471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:08.1376797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:08.1377294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:22:08.1384243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:08.1384661Z dist init r=0, world=1
2023-01-11T21:22:08.1384979Z ok (1.608s)
2023-01-11T21:22:08.1385383Z   test_numel_without_shared_params (__main__.TestFlattenParams)
2023-01-11T21:22:08.1386100Z Tests that numel is preserved after flattening when there are no shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28942
2023-01-11T21:22:08.1387115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:08.1387733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:08.1388540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:08.1388984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:08.1389325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:08.1389830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:22:08.1390229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:08.1390481Z dist init r=0, world=1
2023-01-11T21:22:08.1390658Z ok (1.608s)
2023-01-11T21:22:08.1390890Z   test_output_with_shared_params (__main__.TestFlattenParams)
2023-01-11T21:22:08.1391274Z Tests a forward pass after flattening when there are shared parameters ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28956
2023-01-11T21:22:08.1391808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:08.1392140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:08.1392571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:08.1393019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:08.1393408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:08.1393910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:22:08.1394288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:08.1394547Z dist init r=0, world=1
2023-01-11T21:22:08.1394753Z skip: Need at least 1 CUDA device (1.608s)
2023-01-11T21:22:08.1395016Z   test_output_without_shared_params (__main__.TestFlattenParams)
2023-01-11T21:22:08.1395387Z Tests a forward pass after flattening when there are no shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28970
2023-01-11T21:22:08.1395903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:08.1396232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:08.1396650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:08.1396995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:08.1397328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:08.1397823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:22:08.1398198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:08.1398456Z dist init r=0, world=1
2023-01-11T21:22:08.1398665Z skip: Need at least 1 CUDA device (1.609s)
2023-01-11T21:22:08.1398904Z   test_partial_flattening (__main__.TestFlattenParams)
2023-01-11T21:22:08.1399259Z Tests flattening some submodules but not others. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28984
2023-01-11T21:22:08.1399763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:08.1400093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:08.1400515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:08.1400857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:08.1401189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:08.1401770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:22:08.1402152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:08.1402411Z dist init r=0, world=1
2023-01-11T21:22:08.1402617Z skip: Need at least 1 CUDA device (1.609s)
2023-01-11T21:22:08.1402874Z   test_pnorm_after_step_with_shared_params (__main__.TestFlattenParams)
2023-01-11T21:22:08.1403295Z Tests for parameter Frobenius norm parity after an optimizer step when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28998
2023-01-11T21:22:08.1403817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:08.1404148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:08.1404562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:08.1404911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:08.1405246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:08.1405794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:22:08.1406226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:08.1406491Z dist init r=0, world=1
2023-01-11T21:22:08.1406698Z skip: Need at least 1 CUDA device (1.608s)
2023-01-11T21:22:08.1406835Z 
2023-01-11T21:22:08.1407028Z ----------------------------------------------------------------------
2023-01-11T21:22:08.1407273Z Ran 9 tests in 14.743s
2023-01-11T21:22:08.1407387Z 
2023-01-11T21:22:08.1407461Z OK (skipped=5)
2023-01-11T21:22:08.1407566Z 
2023-01-11T21:22:08.1407654Z Generating XML reports...
2023-01-11T21:22:08.1408105Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params/TEST-TestFlattenParams-20230111212152.xml
2023-01-11T21:22:08.1408372Z 
2023-01-11T21:22:08.1408744Z ##[endgroup]
2023-01-11T21:22:08.1409225Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_flatten_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_flatten_params_stexuohs)
2023-01-11T21:22:08.1409503Z 
2023-01-11T21:22:08.1409713Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2023-01-11 21:22:08.136121]
2023-01-11T21:22:08.1410267Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:22:08.136557]
2023-01-11T21:22:24.8159221Z 
2023-01-11T21:22:24.8160013Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_sharded_grad_scaler
2023-01-11T21:22:24.8161232Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_67aillky)
2023-01-11T21:22:24.8162322Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler
2023-01-11T21:22:24.8162701Z 
2023-01-11T21:22:24.8162810Z Running tests...
2023-01-11T21:22:24.8163261Z ----------------------------------------------------------------------
2023-01-11T21:22:24.8163843Z   test_grad_scaling (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.001s)
2023-01-11T21:22:24.8164500Z   test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.001s)
2023-01-11T21:22:24.8165209Z   test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.002s)
2023-01-11T21:22:24.8166138Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29023
2023-01-11T21:22:24.8166983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29024
2023-01-11T21:22:24.8167573Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29025
2023-01-11T21:22:24.8168163Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29026
2023-01-11T21:22:24.8168999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8169502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8170216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8170801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8171351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:24.8172118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8172677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8173353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8174195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8174849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:24.8175617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8176069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8176645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8177142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8177700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:24.8178539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8179413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8179959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8180582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8181074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8202770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:24.8203668Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8204294Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8204866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8205247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:24.8205605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:24.8205950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:24.8206328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:24.8206759Z dist init r=2, world=4
2023-01-11T21:22:24.8207080Z dist init r=0, world=4
2023-01-11T21:22:24.8207374Z dist init r=1, world=4
2023-01-11T21:22:24.8207658Z dist init r=3, world=4
2023-01-11T21:22:24.8207973Z skip: Need at least 2 CUDA devices (2.178s)
2023-01-11T21:22:24.8208671Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29079
2023-01-11T21:22:24.8210539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29080
2023-01-11T21:22:24.8211145Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29081
2023-01-11T21:22:24.8211536Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29082
2023-01-11T21:22:24.8212138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8212659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8213502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8214120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8214458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:24.8217993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8218434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8218862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8219211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8219548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:24.8220006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8220335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8220762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8221109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8221429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:24.8222311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8222840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8223512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8224157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8224619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:24.8225124Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8225635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8226212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8226733Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8227124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:24.8227462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:24.8227805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:24.8228151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:24.8228413Z dist init r=3, world=4
2023-01-11T21:22:24.8228584Z dist init r=2, world=4
2023-01-11T21:22:24.8228765Z dist init r=1, world=4
2023-01-11T21:22:24.8228941Z dist init r=0, world=4
2023-01-11T21:22:24.8229137Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:22:24.8229587Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29135
2023-01-11T21:22:24.8230068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29136
2023-01-11T21:22:24.8230387Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29137
2023-01-11T21:22:24.8230712Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29138
2023-01-11T21:22:24.8231160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8231490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8231907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8232348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8232735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:24.8233197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8233525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8233950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8234290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8234607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:24.8235075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8235444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8236165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8236525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8236860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:24.8237326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8237638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8238060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8238399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8238735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:24.8239215Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8239735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8240256Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8240765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8241138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:24.8241577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:24.8241920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:24.8242252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:24.8242512Z dist init r=1, world=4
2023-01-11T21:22:24.8242697Z dist init r=0, world=4
2023-01-11T21:22:24.8242874Z dist init r=3, world=4
2023-01-11T21:22:24.8243037Z dist init r=2, world=4
2023-01-11T21:22:24.8243242Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:22:24.8243676Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29191
2023-01-11T21:22:24.8244128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29192
2023-01-11T21:22:24.8244458Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29193
2023-01-11T21:22:24.8244783Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29194
2023-01-11T21:22:24.8245289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8245641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8246078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8246425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8246747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:24.8247219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8247551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8247973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8248307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8248639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:24.8249135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8249634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8249949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8250373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8250718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8251040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:24.8251505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8251832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8252357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8252689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8253023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:24.8253522Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8254027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8254541Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8254930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:24.8255279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:24.8255609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:24.8255946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:24.8256207Z dist init r=0, world=4
2023-01-11T21:22:24.8256388Z dist init r=3, world=4
2023-01-11T21:22:24.8256554Z dist init r=2, world=4
2023-01-11T21:22:24.8256732Z dist init r=1, world=4
2023-01-11T21:22:24.8256938Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:22:24.8257361Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29247
2023-01-11T21:22:24.8257874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29248
2023-01-11T21:22:24.8258199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29249
2023-01-11T21:22:24.8258544Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29250
2023-01-11T21:22:24.8259000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8259329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8259753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8260087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8260423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:24.8260889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8261216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8261634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8262118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8262455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:24.8262913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8263239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8263667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8264009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8264333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:24.8264801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8265129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8265541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8265880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8266212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:24.8266703Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8267213Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8267732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8268250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8268630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:24.8268967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:24.8269303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:24.8269647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:24.8269891Z dist init r=3, world=4
2023-01-11T21:22:24.8270071Z dist init r=0, world=4
2023-01-11T21:22:24.8270250Z dist init r=2, world=4
2023-01-11T21:22:24.8270423Z dist init r=1, world=4
2023-01-11T21:22:24.8270689Z skip: Need at least 2 CUDA devices (1.819s)
2023-01-11T21:22:24.8271152Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29303
2023-01-11T21:22:24.8271612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29304
2023-01-11T21:22:24.8271927Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29305
2023-01-11T21:22:24.8272251Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29306
2023-01-11T21:22:24.8272706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8273033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8273448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8273798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8274133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:24.8274592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8274915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8275338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8275732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8276065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:24.8276516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8276841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8277272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8277618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8277936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:24.8278431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8278925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8279242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8279663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8280007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8280342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:24.8280827Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8281416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8281939Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8282324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:24.8282660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:24.8283000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:24.8283345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:24.8283628Z dist init r=0, world=4
2023-01-11T21:22:24.8283812Z dist init r=1, world=4
2023-01-11T21:22:24.8283993Z dist init r=2, world=4
2023-01-11T21:22:24.8284203Z dist init r=3, world=4
2023-01-11T21:22:24.8284411Z skip: Need at least 2 CUDA devices (1.920s)
2023-01-11T21:22:24.8284859Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29359
2023-01-11T21:22:24.8285337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29360
2023-01-11T21:22:24.8285651Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29361
2023-01-11T21:22:24.8285970Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29362
2023-01-11T21:22:24.8286426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8286758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8287176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8287523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8287860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:24.8288315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8288642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8289071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8289411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8289728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:24.8290196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8290521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8290932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8291271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8291601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:24.8292091Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8292600Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8293094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8293420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8293846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8294176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8294513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:24.8295005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8295505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8295888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:24.8296269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:24.8296643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:24.8296976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:24.8297227Z dist init r=0, world=4
2023-01-11T21:22:24.8297407Z dist init r=1, world=4
2023-01-11T21:22:24.8297573Z dist init r=3, world=4
2023-01-11T21:22:24.8297749Z dist init r=2, world=4
2023-01-11T21:22:24.8297956Z skip: Need at least 2 CUDA devices (1.819s)
2023-01-11T21:22:24.8298387Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29415
2023-01-11T21:22:24.8298840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29416
2023-01-11T21:22:24.8299170Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29417
2023-01-11T21:22:24.8299498Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29418
2023-01-11T21:22:24.8299941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8300268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8300696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8301041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8301359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:24.8301971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8302302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8302719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8303061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8303395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:24.8303860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8304245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8304675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8305067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8305401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:24.8305856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:24.8306184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:24.8306608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:24.8306934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:24.8307267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:24.8307758Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8308283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8308786Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8309377Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:24.8309809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:24.8310160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:24.8310493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:24.8310833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:24.8311093Z dist init r=0, world=4
2023-01-11T21:22:24.8311264Z dist init r=1, world=4
2023-01-11T21:22:24.8311443Z dist init r=3, world=4
2023-01-11T21:22:24.8311622Z dist init r=2, world=4
2023-01-11T21:22:24.8311816Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:22:24.8311957Z 
2023-01-11T21:22:24.8312165Z ----------------------------------------------------------------------
2023-01-11T21:22:24.8312412Z Ran 11 tests in 15.024s
2023-01-11T21:22:24.8312530Z 
2023-01-11T21:22:24.8312605Z OK (skipped=11)
2023-01-11T21:22:24.8312702Z 
2023-01-11T21:22:24.8312795Z Generating XML reports...
2023-01-11T21:22:24.8313281Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20230111212209.xml
2023-01-11T21:22:24.8313962Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20230111212209.xml
2023-01-11T21:22:24.8314280Z 
2023-01-11T21:22:24.8314592Z ##[endgroup]
2023-01-11T21:22:24.8315088Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_67aillky)
2023-01-11T21:22:24.8315376Z 
2023-01-11T21:22:24.8315581Z Running distributed/fsdp/test_fsdp_exec_order ... [2023-01-11 21:22:24.816910]
2023-01-11T21:22:24.8316110Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:22:24.817338]
2023-01-11T21:22:41.1072234Z 
2023-01-11T21:22:41.1075920Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_exec_order
2023-01-11T21:22:41.1081844Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_b_fzenwp)
2023-01-11T21:22:41.1082928Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order
2023-01-11T21:22:41.1083299Z 
2023-01-11T21:22:41.1083429Z Running tests...
2023-01-11T21:22:41.1083977Z ----------------------------------------------------------------------
2023-01-11T21:22:41.1084571Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2023-01-11T21:22:41.1085562Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29482
2023-01-11T21:22:41.1086276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29483
2023-01-11T21:22:41.1086863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29484
2023-01-11T21:22:41.1087386Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29485
2023-01-11T21:22:41.1088224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1088816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1089629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1090284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1090861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:41.1092232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1092882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1093561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1094107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1094631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:41.1095361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1095880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1096584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1097151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1097673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:41.1098446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1098953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1099630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1100197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1100753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:41.1101591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1102718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1103709Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1104606Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1105233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:41.1105861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:41.1106481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:41.1107107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:41.1107575Z dist init r=3, world=4
2023-01-11T21:22:41.1107892Z dist init r=0, world=4
2023-01-11T21:22:41.1108214Z dist init r=2, world=4
2023-01-11T21:22:41.1108523Z dist init r=1, world=4
2023-01-11T21:22:41.1108864Z skip: Need at least 2 CUDA devices (2.082s)
2023-01-11T21:22:41.1109467Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2023-01-11T21:22:41.1110493Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29538
2023-01-11T21:22:41.1111120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29539
2023-01-11T21:22:41.1111648Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29540
2023-01-11T21:22:41.1112187Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29541
2023-01-11T21:22:41.1112955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1113681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1114491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1115068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1115571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:41.1116309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1116834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1117482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1118001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1118514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:41.1119258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1119765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1120423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1120993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1121567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:41.1122318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1122872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1123553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1124054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1124533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:41.1125052Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1125584Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1126106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1126608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1126997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:41.1127353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:41.1127692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:41.1128043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:41.1128298Z dist init r=0, world=4
2023-01-11T21:22:41.1128486Z dist init r=1, world=4
2023-01-11T21:22:41.1128653Z dist init r=3, world=4
2023-01-11T21:22:41.1128830Z dist init r=2, world=4
2023-01-11T21:22:41.1129037Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:22:41.1129372Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2023-01-11T21:22:41.1129953Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29594
2023-01-11T21:22:41.1130364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29595
2023-01-11T21:22:41.1130820Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29596
2023-01-11T21:22:41.1131194Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29597
2023-01-11T21:22:41.1131652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1131985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1132404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1132759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1133096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:41.1133570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1133889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1134322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1134669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1134993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:41.1135460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1135790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1136218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1136547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1136884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:41.1137362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1137694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1138107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1138452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1138785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:41.1139270Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1139794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1140314Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1140839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1141208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:41.1141563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:41.1142113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:41.1142463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:41.1142710Z dist init r=2, world=4
2023-01-11T21:22:41.1142893Z dist init r=3, world=4
2023-01-11T21:22:41.1143075Z dist init r=0, world=4
2023-01-11T21:22:41.1143245Z dist init r=1, world=4
2023-01-11T21:22:41.1143453Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:22:41.1143885Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2023-01-11T21:22:41.1144509Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29650
2023-01-11T21:22:41.1144928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29651
2023-01-11T21:22:41.1145262Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29652
2023-01-11T21:22:41.1145592Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29653
2023-01-11T21:22:41.1146038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1146376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1146811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1147165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1147492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:41.1147966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1148300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1148714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1149063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1149401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:41.1149876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1150194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1150630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1150979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1151302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:41.1151769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1152097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1152525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1152859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1153194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:41.1153695Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1154218Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1154724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1155245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1155638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:41.1155989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:41.1156322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:41.1156704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:41.1156966Z dist init r=1, world=4
2023-01-11T21:22:41.1157176Z dist init r=2, world=4
2023-01-11T21:22:41.1157359Z dist init r=3, world=4
2023-01-11T21:22:41.1157540Z dist init r=0, world=4
2023-01-11T21:22:41.1157733Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:22:41.1158086Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2023-01-11T21:22:41.1158666Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29706
2023-01-11T21:22:41.1159076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29707
2023-01-11T21:22:41.1159392Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29708
2023-01-11T21:22:41.1159716Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29709
2023-01-11T21:22:41.1160176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1160499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1160930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1161360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1161705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:41.1162165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1162499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1162931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1163283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1163608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:41.1164081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1164410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1164824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1165172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1165505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:41.1166000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1166489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1166821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1167250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1167581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1167916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:41.1168413Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1168934Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1169436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1169865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:41.1170250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:41.1170600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:41.1170937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:41.1171193Z dist init r=0, world=4
2023-01-11T21:22:41.1171377Z dist init r=2, world=4
2023-01-11T21:22:41.1171576Z dist init r=1, world=4
2023-01-11T21:22:41.1171761Z dist init r=3, world=4
2023-01-11T21:22:41.1171989Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:22:41.1172326Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2023-01-11T21:22:41.1172912Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29762
2023-01-11T21:22:41.1173320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29763
2023-01-11T21:22:41.1173656Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29764
2023-01-11T21:22:41.1173969Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29765
2023-01-11T21:22:41.1174418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1174749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1175179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1175513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1175855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:41.1176329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1176647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1177080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1177428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1177765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:41.1178218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1178546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1178976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1179306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1179642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:41.1180111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1180438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1180850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1181194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1181529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:41.1182158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1182744Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1183343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1183866Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1184251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:41.1184588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:41.1184933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:41.1185280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:41.1185528Z dist init r=3, world=4
2023-01-11T21:22:41.1185718Z dist init r=2, world=4
2023-01-11T21:22:41.1185901Z dist init r=1, world=4
2023-01-11T21:22:41.1186065Z dist init r=0, world=4
2023-01-11T21:22:41.1186280Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:22:41.1186680Z   test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29818
2023-01-11T21:22:41.1187117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29819
2023-01-11T21:22:41.1187433Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29820
2023-01-11T21:22:41.1187754Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29821
2023-01-11T21:22:41.1188213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1188530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1188963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1189314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1189656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:41.1190114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1190440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1190869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1191213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1191535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:41.1192028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1192532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1192845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1193276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1193622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1193955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:41.1194410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1194736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1195165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1195536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1195918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:41.1196417Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1196938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1197444Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1197830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:41.1198181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:41.1198531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:41.1198934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:41.1199287Z dist init r=1, world=4
2023-01-11T21:22:41.1199522Z dist init r=0, world=4
2023-01-11T21:22:41.1199834Z dist init r=3, world=4
2023-01-11T21:22:41.1200174Z dist init r=2, world=4
2023-01-11T21:22:41.1200445Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:22:41.1200835Z   test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29874
2023-01-11T21:22:41.1201349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29875
2023-01-11T21:22:41.1201687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29876
2023-01-11T21:22:41.1202019Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29877
2023-01-11T21:22:41.1202475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1202811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1203251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1203602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1203926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:22:41.1204400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1204728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1205143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1205492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1205828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:22:41.1206331Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1206820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1207152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1207585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1207932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1208256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:22:41.1208726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:22:41.1209117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:22:41.1209577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:22:41.1209928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:22:41.1210266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:22:41.1210767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1211276Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1211794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:22:41.1212183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:22:41.1212535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:22:41.1212866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:22:41.1213207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:22:41.1213469Z dist init r=0, world=4
2023-01-11T21:22:41.1213639Z dist init r=3, world=4
2023-01-11T21:22:41.1213821Z dist init r=1, world=4
2023-01-11T21:22:41.1214004Z dist init r=2, world=4
2023-01-11T21:22:41.1214200Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:22:41.1214343Z 
2023-01-11T21:22:41.1214549Z ----------------------------------------------------------------------
2023-01-11T21:22:41.1214795Z Ran 8 tests in 14.626s
2023-01-11T21:22:41.1214908Z 
2023-01-11T21:22:41.1214981Z OK (skipped=8)
2023-01-11T21:22:41.1215078Z 
2023-01-11T21:22:41.1215165Z Generating XML reports...
2023-01-11T21:22:41.1215632Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20230111212226.xml
2023-01-11T21:22:41.1215903Z 
2023-01-11T21:22:41.1216244Z ##[endgroup]
2023-01-11T21:22:41.1216696Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_b_fzenwp)
2023-01-11T21:22:41.1216963Z 
2023-01-11T21:22:41.1217161Z Running distributed/test_c10d_spawn_gloo ... [2023-01-11 21:22:41.107778]
2023-01-11T21:22:41.1217707Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:22:41.108305]
2023-01-11T21:23:25.8411014Z 
2023-01-11T21:23:25.8411976Z Expand the folded group to see the log file of distributed/test_c10d_spawn_gloo
2023-01-11T21:23:25.8412957Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_66momomt)
2023-01-11T21:23:25.8413691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63xxdmqs
2023-01-11T21:23:25.8415818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63xxdmqs/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8416790Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelSingleProcessTest testMethod=test_cpu>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]>
2023-01-11T21:23:25.8417450Z test_cpu (__main__.DistributedDataParallelSingleProcessTest)
2023-01-11T21:23:25.8417799Z test_cuda (__main__.DistributedDataParallelSingleProcessTest)
2023-01-11T21:23:25.8418177Z test_rnn (__main__.DistributedDataParallelSingleProcessTest)
2023-01-11T21:23:25.8419163Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_chunk_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_gloo>]>
2023-01-11T21:23:25.8419914Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest)
2023-01-11T21:23:25.8420240Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest)
2023-01-11T21:23:25.8424192Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest)
2023-01-11T21:23:25.8424745Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest)
2023-01-11T21:23:25.8425253Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:23:25.8425602Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:23:25.8427128Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsGloo testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]>
2023-01-11T21:23:25.8428819Z test_all_gather (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T21:23:25.8429348Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T21:23:25.8429783Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T21:23:25.8430362Z test_allreduce (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T21:23:25.8430699Z test_broadcast (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T21:23:25.8430977Z test_gather (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T21:23:25.8431422Z test_reduce (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T21:23:25.8431717Z test_scatter (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T21:23:25.8432358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8432887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8433703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8434346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8434943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvv571wjr
2023-01-11T21:23:25.8435351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvv571wjr/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8435808Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8436008Z 
2023-01-11T21:23:25.8436082Z Running tests...
2023-01-11T21:23:25.8436380Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8436829Z   test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:25.8437418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:23:25.8437800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:23:25.8438061Z ok (0.012s)
2023-01-11T21:23:25.8438162Z 
2023-01-11T21:23:25.8438362Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8438606Z Ran 1 test in 0.012s
2023-01-11T21:23:25.8438733Z 
2023-01-11T21:23:25.8438783Z OK
2023-01-11T21:23:25.8438874Z 
2023-01-11T21:23:25.8438961Z Generating XML reports...
2023-01-11T21:23:25.8439491Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111212244.xml
2023-01-11T21:23:25.8440305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8440626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8441063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8441499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8441839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgg9ds4vu
2023-01-11T21:23:25.8442243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgg9ds4vu/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8442703Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8442898Z 
2023-01-11T21:23:25.8442974Z Running tests...
2023-01-11T21:23:25.8443270Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8443650Z   test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... skip: At least 1 CUDA GPUS needed (0.000s)
2023-01-11T21:23:25.8443883Z 
2023-01-11T21:23:25.8444079Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8444319Z Ran 1 test in 0.001s
2023-01-11T21:23:25.8444419Z 
2023-01-11T21:23:25.8444493Z OK (skipped=1)
2023-01-11T21:23:25.8444599Z 
2023-01-11T21:23:25.8444684Z Generating XML reports...
2023-01-11T21:23:25.8445212Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111212245.xml
2023-01-11T21:23:25.8445790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8451942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8452835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8453490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8454119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogjb3ppp
2023-01-11T21:23:25.8454882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogjb3ppp/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8455733Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8456087Z 
2023-01-11T21:23:25.8456206Z Running tests...
2023-01-11T21:23:25.8456603Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8457247Z   test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... skip: At least 1 CUDA GPUS needed (0.001s)
2023-01-11T21:23:25.8457634Z 
2023-01-11T21:23:25.8457963Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8458387Z Ran 1 test in 0.001s
2023-01-11T21:23:25.8458600Z 
2023-01-11T21:23:25.8458734Z OK (skipped=1)
2023-01-11T21:23:25.8458906Z 
2023-01-11T21:23:25.8459007Z Generating XML reports...
2023-01-11T21:23:25.8459541Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111212247.xml
2023-01-11T21:23:25.8460138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8460467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8460885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8461229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8461842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ncfq2z0
2023-01-11T21:23:25.8462601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ncfq2z0/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8463519Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8463850Z 
2023-01-11T21:23:25.8463929Z Running tests...
2023-01-11T21:23:25.8464237Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8464595Z   test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2023-01-11T21:23:25.8464822Z 
2023-01-11T21:23:25.8465019Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8465256Z Ran 1 test in 0.001s
2023-01-11T21:23:25.8465369Z 
2023-01-11T21:23:25.8465427Z OK (skipped=1)
2023-01-11T21:23:25.8465533Z 
2023-01-11T21:23:25.8465616Z Generating XML reports...
2023-01-11T21:23:25.8466092Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20230111212249.xml
2023-01-11T21:23:25.8466648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8466965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8467391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8467738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8468079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzj_avx9e
2023-01-11T21:23:25.8468465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzj_avx9e/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8468913Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8469109Z 
2023-01-11T21:23:25.8469182Z Running tests...
2023-01-11T21:23:25.8469469Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8469834Z   test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2023-01-11T21:23:25.8470053Z 
2023-01-11T21:23:25.8470247Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8470481Z Ran 1 test in 0.001s
2023-01-11T21:23:25.8470580Z 
2023-01-11T21:23:25.8470650Z OK (skipped=1)
2023-01-11T21:23:25.8470754Z 
2023-01-11T21:23:25.8470866Z Generating XML reports...
2023-01-11T21:23:25.8471339Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20230111212251.xml
2023-01-11T21:23:25.8471885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8472205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8472636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8472987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8473323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzo_oufb0
2023-01-11T21:23:25.8473747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzo_oufb0/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8474199Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8474394Z 
2023-01-11T21:23:25.8474466Z Running tests...
2023-01-11T21:23:25.8474754Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8475116Z   test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2023-01-11T21:23:25.8475333Z 
2023-01-11T21:23:25.8475525Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8475861Z Ran 1 test in 0.001s
2023-01-11T21:23:25.8475982Z 
2023-01-11T21:23:25.8476055Z OK (skipped=1)
2023-01-11T21:23:25.8476162Z 
2023-01-11T21:23:25.8476246Z Generating XML reports...
2023-01-11T21:23:25.8476773Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20230111212253.xml
2023-01-11T21:23:25.8477314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8477642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8478072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8478422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8478753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8utbxtzb
2023-01-11T21:23:25.8479160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8utbxtzb/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8479621Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8479819Z 
2023-01-11T21:23:25.8479880Z Running tests...
2023-01-11T21:23:25.8480183Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8480546Z   test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2023-01-11T21:23:25.8480761Z 
2023-01-11T21:23:25.8480959Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8481181Z Ran 1 test in 0.001s
2023-01-11T21:23:25.8481370Z 
2023-01-11T21:23:25.8481443Z OK (skipped=1)
2023-01-11T21:23:25.8481548Z 
2023-01-11T21:23:25.8481633Z Generating XML reports...
2023-01-11T21:23:25.8482100Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20230111212254.xml
2023-01-11T21:23:25.8482658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8482995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8483430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8483767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8484111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9l8s_3i
2023-01-11T21:23:25.8484509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9l8s_3i/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8484967Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8485149Z 
2023-01-11T21:23:25.8485221Z Running tests...
2023-01-11T21:23:25.8485523Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8485927Z   test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30024
2023-01-11T21:23:25.8490317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30025
2023-01-11T21:23:25.8496028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8496373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8496919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8497256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8497597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hetv0vx
2023-01-11T21:23:25.8497995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hetv0vx/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8498583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8498947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8499379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8499723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8500069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxsw5nbf
2023-01-11T21:23:25.8500457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxsw5nbf/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8500834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:25.8501185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:25.8501460Z skip: Need at least 2 CUDA devices (1.924s)
2023-01-11T21:23:25.8501601Z 
2023-01-11T21:23:25.8502017Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8502264Z Ran 1 test in 1.925s
2023-01-11T21:23:25.8502380Z 
2023-01-11T21:23:25.8502452Z OK (skipped=1)
2023-01-11T21:23:25.8502547Z 
2023-01-11T21:23:25.8502633Z Generating XML reports...
2023-01-11T21:23:25.8503128Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212256.xml
2023-01-11T21:23:25.8503694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8504010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8504441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8504793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8505141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprct1_xbl
2023-01-11T21:23:25.8505528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprct1_xbl/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8505981Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8506177Z 
2023-01-11T21:23:25.8506250Z Running tests...
2023-01-11T21:23:25.8506538Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8506932Z   test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30057
2023-01-11T21:23:25.8507332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30058
2023-01-11T21:23:25.8507781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8508093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8508522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8508867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8509214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9vt3sux
2023-01-11T21:23:25.8509604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9vt3sux/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8510095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8510428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8510845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8511191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8511615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pvegcbt
2023-01-11T21:23:25.8512064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pvegcbt/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8512432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:25.8512784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:25.8513075Z skip: Need at least 2 CUDA devices (1.924s)
2023-01-11T21:23:25.8513212Z 
2023-01-11T21:23:25.8513417Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8513649Z Ran 1 test in 1.924s
2023-01-11T21:23:25.8513761Z 
2023-01-11T21:23:25.8513834Z OK (skipped=1)
2023-01-11T21:23:25.8513941Z 
2023-01-11T21:23:25.8514027Z Generating XML reports...
2023-01-11T21:23:25.8514504Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212300.xml
2023-01-11T21:23:25.8515075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8515415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8515845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8516178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8516525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplc528cu3
2023-01-11T21:23:25.8516928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplc528cu3/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8517365Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8517565Z 
2023-01-11T21:23:25.8517638Z Running tests...
2023-01-11T21:23:25.8517943Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8518357Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30090
2023-01-11T21:23:25.8518753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30091
2023-01-11T21:23:25.8519210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8519539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8519960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8520303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8520645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_sp0a8bf
2023-01-11T21:23:25.8521042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_sp0a8bf/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8521589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8521922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8522350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8522696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8523025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8xvlu96t
2023-01-11T21:23:25.8523423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8xvlu96t/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8523797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:25.8524132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:25.8524463Z skip: Need at least 2 CUDA devices (1.924s)
2023-01-11T21:23:25.8524599Z 
2023-01-11T21:23:25.8524835Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8525076Z Ran 1 test in 1.925s
2023-01-11T21:23:25.8525176Z 
2023-01-11T21:23:25.8525248Z OK (skipped=1)
2023-01-11T21:23:25.8525354Z 
2023-01-11T21:23:25.8525438Z Generating XML reports...
2023-01-11T21:23:25.8525927Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212304.xml
2023-01-11T21:23:25.8526472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8526800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8527231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8527579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8527914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftho44d6
2023-01-11T21:23:25.8528314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftho44d6/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8528763Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8528960Z 
2023-01-11T21:23:25.8529031Z Running tests...
2023-01-11T21:23:25.8529323Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8529722Z   test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30123
2023-01-11T21:23:25.8530125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30124
2023-01-11T21:23:25.8530565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8530897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8531326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8531671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8532003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqys_o_0q
2023-01-11T21:23:25.8532398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqys_o_0q/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8532883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8533199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8533620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8533965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8534310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4zk_rst8
2023-01-11T21:23:25.8534697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4zk_rst8/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8535071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:25.8535416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:25.8535694Z skip: Need at least 2 CUDA devices (1.925s)
2023-01-11T21:23:25.8535828Z 
2023-01-11T21:23:25.8536030Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8536266Z Ran 1 test in 1.925s
2023-01-11T21:23:25.8536377Z 
2023-01-11T21:23:25.8536449Z OK (skipped=1)
2023-01-11T21:23:25.8536543Z 
2023-01-11T21:23:25.8536625Z Generating XML reports...
2023-01-11T21:23:25.8537114Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212308.xml
2023-01-11T21:23:25.8537745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8538074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8538491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8538836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8539184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64fudn28
2023-01-11T21:23:25.8539575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64fudn28/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8540027Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8540220Z 
2023-01-11T21:23:25.8540295Z Running tests...
2023-01-11T21:23:25.8540602Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8540990Z   test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30156
2023-01-11T21:23:25.8541388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30157
2023-01-11T21:23:25.8542032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8542349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8542786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8543136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8543484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xnch__7
2023-01-11T21:23:25.8543874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xnch__7/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8544363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8544688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8545110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8545438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8545776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl65myls1
2023-01-11T21:23:25.8546177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl65myls1/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8546545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:25.8546890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:25.8547177Z skip: Need at least 2 CUDA devices (1.924s)
2023-01-11T21:23:25.8547312Z 
2023-01-11T21:23:25.8547513Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8547741Z Ran 1 test in 1.924s
2023-01-11T21:23:25.8547852Z 
2023-01-11T21:23:25.8547921Z OK (skipped=1)
2023-01-11T21:23:25.8548027Z 
2023-01-11T21:23:25.8548110Z Generating XML reports...
2023-01-11T21:23:25.8548583Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212311.xml
2023-01-11T21:23:25.8549142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8549471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8549895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8550300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8550711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnrfsuap1
2023-01-11T21:23:25.8551124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnrfsuap1/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8551573Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8551768Z 
2023-01-11T21:23:25.8551841Z Running tests...
2023-01-11T21:23:25.8552146Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8552549Z   test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30189
2023-01-11T21:23:25.8552935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30190
2023-01-11T21:23:25.8553394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8553801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8554231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8554614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8554960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57od_5kw
2023-01-11T21:23:25.8555357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57od_5kw/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8555829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8556154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8556581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8556929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8557257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_iq4kkdk
2023-01-11T21:23:25.8557654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_iq4kkdk/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8558028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:25.8558363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:25.8558647Z skip: Need at least 2 CUDA devices (1.926s)
2023-01-11T21:23:25.8558785Z 
2023-01-11T21:23:25.8558988Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8559227Z Ran 1 test in 1.926s
2023-01-11T21:23:25.8559328Z 
2023-01-11T21:23:25.8559401Z OK (skipped=1)
2023-01-11T21:23:25.8559507Z 
2023-01-11T21:23:25.8559591Z Generating XML reports...
2023-01-11T21:23:25.8560080Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212315.xml
2023-01-11T21:23:25.8560628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8560955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8561445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8561796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8562129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzucor9eg
2023-01-11T21:23:25.8562533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzucor9eg/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8562989Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8563244Z 
2023-01-11T21:23:25.8563314Z Running tests...
2023-01-11T21:23:25.8563606Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8564044Z   test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30222
2023-01-11T21:23:25.8564450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30223
2023-01-11T21:23:25.8564896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8565231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8565666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8566018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8566353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51cjomyu
2023-01-11T21:23:25.8566767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51cjomyu/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8567261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8567593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8568011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8568358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8568706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa7sc4r2x
2023-01-11T21:23:25.8569094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa7sc4r2x/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8569476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:25.8569828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:25.8570124Z skip: Need at least 2 CUDA devices (1.925s)
2023-01-11T21:23:25.8570253Z 
2023-01-11T21:23:25.8570462Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8570704Z Ran 1 test in 1.925s
2023-01-11T21:23:25.8570819Z 
2023-01-11T21:23:25.8570892Z OK (skipped=1)
2023-01-11T21:23:25.8571001Z 
2023-01-11T21:23:25.8571074Z Generating XML reports...
2023-01-11T21:23:25.8571570Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212319.xml
2023-01-11T21:23:25.8572134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8572471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8572890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8573242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8573595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsri2bucd
2023-01-11T21:23:25.8573991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsri2bucd/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8574449Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T21:23:25.8574646Z 
2023-01-11T21:23:25.8574720Z Running tests...
2023-01-11T21:23:25.8575023Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8575410Z   test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30255
2023-01-11T21:23:25.8575811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30256
2023-01-11T21:23:25.8576300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8576663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8577082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8577425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8577769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpirn9up7m
2023-01-11T21:23:25.8578157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpirn9up7m/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8578643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:25.8578981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:25.8579411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:25.8579751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:25.8580097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpygfeufe_
2023-01-11T21:23:25.8580498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpygfeufe_/_remote_module_non_scriptable.py
2023-01-11T21:23:25.8580865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:25.8581212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:25.8581494Z skip: Need at least 2 CUDA devices (1.925s)
2023-01-11T21:23:25.8581630Z 
2023-01-11T21:23:25.8582051Z ----------------------------------------------------------------------
2023-01-11T21:23:25.8582399Z Ran 1 test in 1.926s
2023-01-11T21:23:25.8582561Z 
2023-01-11T21:23:25.8582663Z OK (skipped=1)
2023-01-11T21:23:25.8582829Z 
2023-01-11T21:23:25.8582947Z Generating XML reports...
2023-01-11T21:23:25.8583469Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212323.xml
2023-01-11T21:23:25.8583762Z 
2023-01-11T21:23:25.8584207Z ##[endgroup]
2023-01-11T21:23:25.8584648Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_66momomt)
2023-01-11T21:23:25.8584899Z 
2023-01-11T21:23:25.8585134Z Running distributed/checkpoint/test_file_system_checkpoint_cpu ... [2023-01-11 21:23:25.841751]
2023-01-11T21:23:25.8585695Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:23:25.842097]
2023-01-11T21:23:45.6542164Z 
2023-01-11T21:23:45.6542695Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint_cpu
2023-01-11T21:23:45.6543833Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_u1xzx_bm)
2023-01-11T21:23:45.6544898Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu
2023-01-11T21:23:45.6545250Z 
2023-01-11T21:23:45.6545359Z Running tests...
2023-01-11T21:23:45.6545840Z ----------------------------------------------------------------------
2023-01-11T21:23:45.6546503Z   test_load_rowwise_to_colwise_thread_count_1 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30288
2023-01-11T21:23:45.6547177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30289
2023-01-11T21:23:45.6547912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6548691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6549535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6550098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6550632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:45.6551451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6552013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6552819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6553487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6554111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:45.6554703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:45.6555303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:45.6556238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6557187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6558865Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6559942Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6561441Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6562468Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6562847Z ok (1.981s)
2023-01-11T21:23:45.6563504Z   test_load_rowwise_to_colwise_thread_count_2 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30316
2023-01-11T21:23:45.6564250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30317
2023-01-11T21:23:45.6565124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6565644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6566312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6566860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6567371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:45.6568097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6568610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6569308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6570002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6570522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:45.6571217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:45.6571852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:45.6572667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6573572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6575096Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6576068Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6577619Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6578687Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6579019Z ok (1.714s)
2023-01-11T21:23:45.6579689Z   test_load_with_different_shard_plan_thread_count_1 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30346
2023-01-11T21:23:45.6580494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30347
2023-01-11T21:23:45.6581286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6581985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6582728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6583276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6583811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:45.6584560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6585099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6585781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6586303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6586813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:45.6587381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:45.6587931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:45.6588725Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6589521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6590988Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6592064Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6593491Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6594298Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6594492Z ok (1.915s)
2023-01-11T21:23:45.6594860Z   test_load_with_different_shard_plan_thread_count_2 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30374
2023-01-11T21:23:45.6595303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30375
2023-01-11T21:23:45.6595775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6596102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6596539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6596892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6597206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:45.6597670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6598002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6598435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6598769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6599093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:45.6599463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:45.6599836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:45.6600322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6600847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6601834Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6602398Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6603191Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6603747Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6603953Z ok (1.915s)
2023-01-11T21:23:45.6604300Z   test_save_load_bytes_thread_count_1 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30426
2023-01-11T21:23:45.6604707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30427
2023-01-11T21:23:45.6605168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6605564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6606041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6606374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6606703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:45.6607170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6607487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6607924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6608270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6608594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:45.6608952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:45.6609326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:45.6609825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6610344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6610623Z ok (1.713s)
2023-01-11T21:23:45.6610969Z   test_save_load_bytes_thread_count_2 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30454
2023-01-11T21:23:45.6611390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30455
2023-01-11T21:23:45.6611829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6612167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6612600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6612950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6613259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:45.6613720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6614048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6614477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6614809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6615135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:45.6615501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:45.6615856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:45.6616349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6616868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6617158Z ok (1.612s)
2023-01-11T21:23:45.6617518Z   test_switch_between_sharded_tensor_to_tensor_thread_count_1 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30484
2023-01-11T21:23:45.6617957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30485
2023-01-11T21:23:45.6618447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6618809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6619232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6619577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6619904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:45.6620352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6620683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6621112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6621459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6621944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:45.6622314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:45.6622681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:45.6623166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6623685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6624590Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6625154Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6625965Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6626518Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6626706Z ok (1.915s)
2023-01-11T21:23:45.6627078Z   test_switch_between_sharded_tensor_to_tensor_thread_count_2 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30512
2023-01-11T21:23:45.6627520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30513
2023-01-11T21:23:45.6627966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6628296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6628725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6629070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6629380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:45.6629844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6630172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6630589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6631017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6631390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:45.6631763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:45.6632118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:45.6632613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6633136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6634042Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6634598Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6635411Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6635965Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6636171Z ok (1.915s)
2023-01-11T21:23:45.6637102Z   test_read_write_only_tensor_thread_count_1 (__main__.TestDistributedStateDictSaveLoad) ... /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6637745Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6637942Z ok (0.020s)
2023-01-11T21:23:45.6638228Z   test_read_write_only_tensor_thread_count_2 (__main__.TestDistributedStateDictSaveLoad) ... ok (0.012s)
2023-01-11T21:23:45.6638743Z   test_read_write_shard_tensor_thread_count_1 (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30573
2023-01-11T21:23:45.6639205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30574
2023-01-11T21:23:45.6639661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6639994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6640415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6640769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6641097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:45.6641664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6641985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6642416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6642761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6643090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:45.6643490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:45.6643910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:45.6644416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6644924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6645823Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6646386Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6647197Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6647756Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6647944Z ok (1.713s)
2023-01-11T21:23:45.6648350Z   test_read_write_shard_tensor_thread_count_2 (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30601
2023-01-11T21:23:45.6648827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30602
2023-01-11T21:23:45.6649286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6649607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6650043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6650392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6650703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:45.6651169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:23:45.6651497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:45.6651927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:45.6652261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:45.6652583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:45.6652951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:45.6653322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:45.6653803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6654324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:45.6655228Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6655826Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6656645Z /opt/conda/lib/python3.7/site-packages/torch/distributed/checkpoint/filesystem.py:110: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T21:23:45.6657205Z   if tensor.storage().size() != tensor.numel():
2023-01-11T21:23:45.6657407Z ok (1.713s)
2023-01-11T21:23:45.6657512Z 
2023-01-11T21:23:45.6657717Z ----------------------------------------------------------------------
2023-01-11T21:23:45.6657952Z Ran 12 tests in 18.139s
2023-01-11T21:23:45.6658066Z 
2023-01-11T21:23:45.6658128Z OK
2023-01-11T21:23:45.6658220Z 
2023-01-11T21:23:45.6658306Z Generating XML reports...
2023-01-11T21:23:45.6658837Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20230111212327.xml
2023-01-11T21:23:45.6659551Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20230111212327.xml
2023-01-11T21:23:45.6660352Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20230111212327.xml
2023-01-11T21:23:45.6660729Z 
2023-01-11T21:23:45.6661057Z ##[endgroup]
2023-01-11T21:23:45.6661573Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_u1xzx_bm)
2023-01-11T21:23:45.6662040Z 
2023-01-11T21:23:45.6662238Z Running distributed/test_pg_wrapper ... [2023-01-11 21:23:45.654847]
2023-01-11T21:23:45.6662774Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:23:45.655205]
2023-01-11T21:24:28.3093370Z 
2023-01-11T21:24:28.3094360Z Expand the folded group to see the log file of distributed/test_pg_wrapper
2023-01-11T21:24:28.3095505Z ##[group]PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-test_pg_wrapper_w67phow_)
2023-01-11T21:24:28.3095995Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:24:28.3096355Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:24:28.3100250Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2023-01-11T21:24:28.3102477Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T21:24:28.3103087Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T21:24:28.3103661Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T21:24:28.3104000Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T21:24:28.3104509Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T21:24:28.3104849Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T21:24:28.3105162Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T21:24:28.3105686Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T21:24:28.3106098Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T21:24:28.3106890Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2023-01-11T21:24:28.3107659Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest)
2023-01-11T21:24:28.3108123Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2023-01-11T21:24:28.3108466Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2023-01-11T21:24:28.3134465Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2023-01-11T21:24:28.3135166Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2023-01-11T21:24:28.3135817Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3136451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3136881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3137616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3138262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3138556Z 
2023-01-11T21:24:28.3138693Z Running tests...
2023-01-11T21:24:28.3139205Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3139953Z   test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30652
2023-01-11T21:24:28.3140433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30653
2023-01-11T21:24:28.3140769Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30654
2023-01-11T21:24:28.3141084Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30655
2023-01-11T21:24:28.3141544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3142068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3142617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3143084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3143550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:28.3144436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3144920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3145345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3145692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3146018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:28.3146472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3146803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3147233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3147743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3148140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:28.3148608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3148940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3149359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3149703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3150030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:28.3150396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:24:28.3150756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:24:28.3151127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:24:28.3151490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:24:28.3151984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3152492Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3153015Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3153534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3153919Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2023-01-11T21:24:28.3154257Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2023-01-11T21:24:28.3154699Z [E ProcessGroupGloo.cpp:138] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2023-01-11T21:24:28.3155214Z [E ProcessGroupGloo.cpp:138] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2023-01-11T21:24:28.3155537Z ok (2.182s)
2023-01-11T21:24:28.3155643Z 
2023-01-11T21:24:28.3155846Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3156093Z Ran 1 test in 2.182s
2023-01-11T21:24:28.3156208Z 
2023-01-11T21:24:28.3156269Z OK
2023-01-11T21:24:28.3156348Z 
2023-01-11T21:24:28.3156435Z Generating XML reports...
2023-01-11T21:24:28.3156910Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212348.xml
2023-01-11T21:24:28.3157424Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3157879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3158219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3158652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3159003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3159168Z 
2023-01-11T21:24:28.3159228Z Running tests...
2023-01-11T21:24:28.3159532Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3159954Z   test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30743
2023-01-11T21:24:28.3160411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30744
2023-01-11T21:24:28.3160763Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30745
2023-01-11T21:24:28.3161090Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30746
2023-01-11T21:24:28.3161629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3161949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3162382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3162735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3163065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:28.3163514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3163849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3169472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3170134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3170771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:28.3171603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3172123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3175129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3175483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3175820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:28.3176287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3176608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3177041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3177384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3177700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:28.3178067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:24:28.3178436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:24:28.3178806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:24:28.3179157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:24:28.3179650Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3180172Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3180690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3181195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3181483Z ok (2.276s)
2023-01-11T21:24:28.3181588Z 
2023-01-11T21:24:28.3182041Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3182477Z Ran 1 test in 2.276s
2023-01-11T21:24:28.3182648Z 
2023-01-11T21:24:28.3182764Z OK
2023-01-11T21:24:28.3182939Z 
2023-01-11T21:24:28.3183106Z Generating XML reports...
2023-01-11T21:24:28.3183982Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212351.xml
2023-01-11T21:24:28.3184874Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3185558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3186128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3186842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3187407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3187705Z 
2023-01-11T21:24:28.3187835Z Running tests...
2023-01-11T21:24:28.3188349Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3189120Z   test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30834
2023-01-11T21:24:28.3189898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30835
2023-01-11T21:24:28.3190438Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30836
2023-01-11T21:24:28.3190956Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30837
2023-01-11T21:24:28.3191717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3192309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3193118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3193748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3194321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:28.3195097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3195650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3196437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3197067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3197393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:28.3197843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3198172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3198602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3199117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3199654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:28.3200370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3200848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3201615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3202169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3202691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:28.3203301Z skip: Need at least 4 CUDA devices (1.878s)
2023-01-11T21:24:28.3203554Z 
2023-01-11T21:24:28.3204166Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3204673Z Ran 1 test in 1.878s
2023-01-11T21:24:28.3204881Z 
2023-01-11T21:24:28.3205007Z OK (skipped=1)
2023-01-11T21:24:28.3205204Z 
2023-01-11T21:24:28.3205368Z Generating XML reports...
2023-01-11T21:24:28.3206061Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212355.xml
2023-01-11T21:24:28.3206580Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3207044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3207364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3207797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3208150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3208313Z 
2023-01-11T21:24:28.3208388Z Running tests...
2023-01-11T21:24:28.3208678Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3209115Z   test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30889
2023-01-11T21:24:28.3209546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30890
2023-01-11T21:24:28.3209879Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30891
2023-01-11T21:24:28.3210197Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30892
2023-01-11T21:24:28.3210647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3210988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3211405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3211751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3212072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:28.3212537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3212852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3213284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3213628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3213941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:28.3214405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3214734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3215164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3215532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3215854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:28.3216318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3216646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3217062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3217455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3217783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:28.3218103Z skip: Need at least 4 CUDA devices (1.978s)
2023-01-11T21:24:28.3218248Z 
2023-01-11T21:24:28.3218450Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3218690Z Ran 1 test in 1.978s
2023-01-11T21:24:28.3218810Z 
2023-01-11T21:24:28.3218883Z OK (skipped=1)
2023-01-11T21:24:28.3218979Z 
2023-01-11T21:24:28.3219067Z Generating XML reports...
2023-01-11T21:24:28.3219544Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212359.xml
2023-01-11T21:24:28.3220056Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3220511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3220846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3221279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3221623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3221946Z 
2023-01-11T21:24:28.3222021Z Running tests...
2023-01-11T21:24:28.3222334Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3222780Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30944
2023-01-11T21:24:28.3223198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30945
2023-01-11T21:24:28.3223528Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30946
2023-01-11T21:24:28.3223858Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30947
2023-01-11T21:24:28.3224325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3224646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3225080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3225427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3225747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:28.3226195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3226526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3226961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3227300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3227624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:28.3228086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3228416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3228832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3229175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3229495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:28.3229949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3230279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3230795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3231183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3231499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:28.3231868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:24:28.3232247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:24:28.3232615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:24:28.3233100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3233503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:24:28.3233995Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3234508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3235031Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3235426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:24:28.3235786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:24:28.3236137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T21:24:28.3236496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:24:28.3236983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:24:28.3237496Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:24:28.3237999Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:24:28.3238515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:24:28.3238803Z ok (2.285s)
2023-01-11T21:24:28.3238906Z 
2023-01-11T21:24:28.3239106Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3239337Z Ran 1 test in 2.285s
2023-01-11T21:24:28.3239453Z 
2023-01-11T21:24:28.3239517Z OK
2023-01-11T21:24:28.3239611Z 
2023-01-11T21:24:28.3239694Z Generating XML reports...
2023-01-11T21:24:28.3240160Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212402.xml
2023-01-11T21:24:28.3240680Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3241228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3241564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3241984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3242331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3242497Z 
2023-01-11T21:24:28.3242571Z Running tests...
2023-01-11T21:24:28.3242868Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3243290Z   test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31047
2023-01-11T21:24:28.3243770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31048
2023-01-11T21:24:28.3244140Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31049
2023-01-11T21:24:28.3244459Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31050
2023-01-11T21:24:28.3244913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3245245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3245667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3246008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3246333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:28.3246794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3247107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3247533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3247882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3248203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:28.3248651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3248980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3249411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3249748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3250075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:28.3250538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3250861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3251275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3251624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3251944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:28.3252320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:24:28.3252687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:24:28.3253053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:24:28.3253416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:24:28.3253897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3254413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3254934Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3255447Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3255719Z ok (2.179s)
2023-01-11T21:24:28.3255819Z 
2023-01-11T21:24:28.3256018Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3256307Z Ran 1 test in 2.179s
2023-01-11T21:24:28.3256420Z 
2023-01-11T21:24:28.3256470Z OK
2023-01-11T21:24:28.3256567Z 
2023-01-11T21:24:28.3256653Z Generating XML reports...
2023-01-11T21:24:28.3257165Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212406.xml
2023-01-11T21:24:28.3257683Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3258138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3258473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3258914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3259259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3259412Z 
2023-01-11T21:24:28.3259488Z Running tests...
2023-01-11T21:24:28.3259791Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3260219Z   test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31138
2023-01-11T21:24:28.3260624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31139
2023-01-11T21:24:28.3260953Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31140
2023-01-11T21:24:28.3261277Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31141
2023-01-11T21:24:28.3261888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3262219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3262661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3263014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3263336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:28.3263802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3264133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3264566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3264897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3265222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:28.3265687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3266016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3266435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3266785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3267109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:28.3267557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3267884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3268313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3268661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3268973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:28.3269344Z skip: Need at least 4 CUDA devices (1.979s)
2023-01-11T21:24:28.3269480Z 
2023-01-11T21:24:28.3269686Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3269962Z Ran 1 test in 1.980s
2023-01-11T21:24:28.3270076Z 
2023-01-11T21:24:28.3270152Z OK (skipped=1)
2023-01-11T21:24:28.3270265Z 
2023-01-11T21:24:28.3270355Z Generating XML reports...
2023-01-11T21:24:28.3270834Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212410.xml
2023-01-11T21:24:28.3271333Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3271795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3272127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3272548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3272902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3273065Z 
2023-01-11T21:24:28.3273145Z Running tests...
2023-01-11T21:24:28.3273455Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3273877Z   test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31193
2023-01-11T21:24:28.3274311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31194
2023-01-11T21:24:28.3274645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31195
2023-01-11T21:24:28.3274962Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31196
2023-01-11T21:24:28.3275409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3275747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3276180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3276512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3276832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:28.3277292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3277616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3278027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3278372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3278697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:28.3279147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3279481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3279910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3280254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3280564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:28.3281027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3281449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3281871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3282277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3282603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:28.3282925Z skip: Need at least 4 CUDA devices (1.973s)
2023-01-11T21:24:28.3283068Z 
2023-01-11T21:24:28.3283258Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3283501Z Ran 1 test in 1.974s
2023-01-11T21:24:28.3283613Z 
2023-01-11T21:24:28.3283683Z OK (skipped=1)
2023-01-11T21:24:28.3283788Z 
2023-01-11T21:24:28.3283863Z Generating XML reports...
2023-01-11T21:24:28.3284339Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212414.xml
2023-01-11T21:24:28.3284854Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3285319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3285639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3286071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3286428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3286593Z 
2023-01-11T21:24:28.3286667Z Running tests...
2023-01-11T21:24:28.3286957Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3287388Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31248
2023-01-11T21:24:28.3287819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31249
2023-01-11T21:24:28.3288138Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31250
2023-01-11T21:24:28.3288464Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31251
2023-01-11T21:24:28.3288914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3289249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3289667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3290008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3290333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:28.3290782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3291115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3291536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3291885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3292197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:28.3292658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3292985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3293409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3293739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3294058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:28.3294520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3294832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3295315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3295691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3296013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:28.3296367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:24:28.3296739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:24:28.3297109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:24:28.3297462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:24:28.3297957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3298485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3299006Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3299508Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:24:28.3299896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:24:28.3300264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:24:28.3300625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:24:28.3300971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T21:24:28.3301466Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:24:28.3302149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:24:28.3302667Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:24:28.3303165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:24:28.3303456Z ok (2.292s)
2023-01-11T21:24:28.3303565Z 
2023-01-11T21:24:28.3303769Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3304017Z Ran 1 test in 2.292s
2023-01-11T21:24:28.3304118Z 
2023-01-11T21:24:28.3304181Z OK
2023-01-11T21:24:28.3304274Z 
2023-01-11T21:24:28.3304360Z Generating XML reports...
2023-01-11T21:24:28.3304849Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212417.xml
2023-01-11T21:24:28.3305391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3305723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3306154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3306498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3306651Z 
2023-01-11T21:24:28.3306722Z Running tests...
2023-01-11T21:24:28.3307030Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3307553Z   test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3307921Z skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:24:28.3308153Z 
2023-01-11T21:24:28.3308353Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3308589Z Ran 1 test in 0.000s
2023-01-11T21:24:28.3308745Z 
2023-01-11T21:24:28.3308822Z OK (skipped=1)
2023-01-11T21:24:28.3308917Z 
2023-01-11T21:24:28.3309001Z Generating XML reports...
2023-01-11T21:24:28.3309477Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111212421.xml
2023-01-11T21:24:28.3310031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3310352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3310783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3311129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3311292Z 
2023-01-11T21:24:28.3311368Z Running tests...
2023-01-11T21:24:28.3311663Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3312201Z   test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3312592Z skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:24:28.3312745Z 
2023-01-11T21:24:28.3312945Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3313176Z Ran 1 test in 0.000s
2023-01-11T21:24:28.3313290Z 
2023-01-11T21:24:28.3313364Z OK (skipped=1)
2023-01-11T21:24:28.3313472Z 
2023-01-11T21:24:28.3313556Z Generating XML reports...
2023-01-11T21:24:28.3314012Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111212423.xml
2023-01-11T21:24:28.3314566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3314901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3315335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3315666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3315831Z 
2023-01-11T21:24:28.3315902Z Running tests...
2023-01-11T21:24:28.3316200Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3316743Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3317148Z skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:24:28.3317301Z 
2023-01-11T21:24:28.3317492Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3317732Z Ran 1 test in 0.000s
2023-01-11T21:24:28.3317830Z 
2023-01-11T21:24:28.3317903Z OK (skipped=1)
2023-01-11T21:24:28.3318006Z 
2023-01-11T21:24:28.3318090Z Generating XML reports...
2023-01-11T21:24:28.3318567Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111212424.xml
2023-01-11T21:24:28.3319103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3319435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3319860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3320208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3320374Z 
2023-01-11T21:24:28.3320435Z Running tests...
2023-01-11T21:24:28.3320735Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3321382Z   test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3321821Z skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:24:28.3321963Z 
2023-01-11T21:24:28.3322161Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3322405Z Ran 1 test in 0.000s
2023-01-11T21:24:28.3322516Z 
2023-01-11T21:24:28.3322588Z OK (skipped=1)
2023-01-11T21:24:28.3322696Z 
2023-01-11T21:24:28.3322767Z Generating XML reports...
2023-01-11T21:24:28.3323238Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111212426.xml
2023-01-11T21:24:28.3323796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:28.3324128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:28.3324546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:28.3324894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:28.3325057Z 
2023-01-11T21:24:28.3325129Z Running tests...
2023-01-11T21:24:28.3325420Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3325977Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T21:24:28.3326377Z skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:24:28.3326526Z 
2023-01-11T21:24:28.3326720Z ----------------------------------------------------------------------
2023-01-11T21:24:28.3326950Z Ran 1 test in 0.000s
2023-01-11T21:24:28.3327061Z 
2023-01-11T21:24:28.3327134Z OK (skipped=1)
2023-01-11T21:24:28.3327242Z 
2023-01-11T21:24:28.3327329Z Generating XML reports...
2023-01-11T21:24:28.3327786Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111212427.xml
2023-01-11T21:24:28.3328066Z 
2023-01-11T21:24:28.3328610Z ##[endgroup]
2023-01-11T21:24:28.3329043Z FINISHED PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-test_pg_wrapper_w67phow_)
2023-01-11T21:24:28.3329287Z 
2023-01-11T21:24:28.3329504Z Running distributed/elastic/multiprocessing/api_test ... [2023-01-11 21:24:28.310364]
2023-01-11T21:24:28.3330056Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:24:28.310722]
2023-01-11T21:24:50.2060721Z 
2023-01-11T21:24:50.2061581Z Expand the folded group to see the log file of distributed/elastic/multiprocessing/api_test
2023-01-11T21:24:50.2062559Z ##[group]PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_oy3j4zhi)
2023-01-11T21:24:50.2064579Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test
2023-01-11T21:24:50.2065037Z 
2023-01-11T21:24:50.2065173Z Running tests...
2023-01-11T21:24:50.2065690Z ----------------------------------------------------------------------
2023-01-11T21:24:50.2066175Z   test_get_failures (__main__.RunProcResultsTest) ... ok (0.258s)
2023-01-11T21:24:50.2066685Z   test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s)
2023-01-11T21:24:50.2067243Z   test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.001s)
2023-01-11T21:24:50.2067718Z   test_binary (__main__.StartProcessesListTest) ... hello stderr from 0
2023-01-11T21:24:50.2068109Z hello stdout from 0
2023-01-11T21:24:50.2068405Z hello stderr from 1
2023-01-11T21:24:50.2068711Z hello stdout from 1
2023-01-11T21:24:50.2069015Z ok (0.116s)
2023-01-11T21:24:50.2069721Z   test_binary_exit (__main__.StartProcessesListTest) ... bar stderr from 1
2023-01-11T21:24:50.2070140Z bar stdout from 1
2023-01-11T21:24:50.2070688Z failed (exitcode: 138) local_rank: 0 (pid: 31402) of binary: distributed/elastic/multiprocessing/bin/echo1.py
2023-01-11T21:24:50.2071082Z ok (0.117s)
2023-01-11T21:24:50.2071405Z   test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.009s)
2023-01-11T21:24:50.2072024Z   test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last):
2023-01-11T21:24:50.2072469Z   File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2023-01-11T21:24:50.2072744Z     raise RuntimeError(f"raised from {rank}")
2023-01-11T21:24:50.2072948Z RuntimeError: raised from 0
2023-01-11T21:24:50.2073130Z bar from 1
2023-01-11T21:24:50.2073406Z failed (exitcode: 1) local_rank: 0 (pid: 31405) of binary: distributed/elastic/multiprocessing/bin/echo2.py
2023-01-11T21:24:50.2073653Z ok (0.114s)
2023-01-11T21:24:50.2073909Z   test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1
2023-01-11T21:24:50.2074177Z [trainer1]:world stderr from 1
2023-01-11T21:24:50.2074346Z ok (1.118s)
2023-01-11T21:24:50.2074597Z   test_function (__main__.StartProcessesListTest) ... [trainer0]:hello stdout from 0
2023-01-11T21:24:50.2075092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2075411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2075842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2076188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2076621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2076936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2077362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2077707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2077941Z hello stdout from 1
2023-01-11T21:24:50.2078111Z hello stdout from 0
2023-01-11T21:24:50.2078287Z hello stderr from 1
2023-01-11T21:24:50.2078460Z hello stderr from 0
2023-01-11T21:24:50.2078646Z Closing process 31412 via signal SIGTERM
2023-01-11T21:24:50.2078838Z ok (1.685s)
2023-01-11T21:24:50.2079321Z   test_function_large_ret_val (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2079701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2080127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2080471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2080892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2081292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2081796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2082136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2082564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2082889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2083297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2083631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2084139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2084504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2084918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2085259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2085511Z Closing process 31432 via signal SIGTERM
2023-01-11T21:24:50.2085719Z Closing process 31434 via signal SIGTERM
2023-01-11T21:24:50.2085911Z ok (2.097s)
2023-01-11T21:24:50.2086130Z   test_function_raise (__main__.StartProcessesListTest)
2023-01-11T21:24:50.2086636Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2087013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2087441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2087784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2088202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2088528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2088950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2089296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2089588Z failed (exitcode: 1) local_rank: 0 (pid: 31472) of fn: echo2 (start_method: spawn)
2023-01-11T21:24:50.2089844Z Traceback (most recent call last):
2023-01-11T21:24:50.2090248Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2023-01-11T21:24:50.2090547Z     self._pc.join(-1)
2023-01-11T21:24:50.2090900Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2023-01-11T21:24:50.2091242Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2023-01-11T21:24:50.2091568Z torch.multiprocessing.spawn.ProcessRaisedException: 
2023-01-11T21:24:50.2091765Z 
2023-01-11T21:24:50.2091928Z -- Process 0 terminated with the following error:
2023-01-11T21:24:50.2092168Z Traceback (most recent call last):
2023-01-11T21:24:50.2092539Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2023-01-11T21:24:50.2092788Z     fn(i, *args)
2023-01-11T21:24:50.2093163Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2023-01-11T21:24:50.2093453Z     ret = record(fn)(*args_)
2023-01-11T21:24:50.2093848Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2023-01-11T21:24:50.2094153Z     return f(*args, **kwargs)
2023-01-11T21:24:50.2094443Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2023-01-11T21:24:50.2094722Z     raise RuntimeError(msg)
2023-01-11T21:24:50.2094901Z RuntimeError: hello
2023-01-11T21:24:50.2095018Z 
2023-01-11T21:24:50.2095085Z ok (1.760s)
2023-01-11T21:24:50.2095326Z   test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s)
2023-01-11T21:24:50.2095620Z   test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.001s)
2023-01-11T21:24:50.2095968Z   test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 31492 via signal SIGTERM
2023-01-11T21:24:50.2096237Z ok (0.009s)
2023-01-11T21:24:50.2096688Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2023-01-11T21:24:50.2097078Z Traceback (most recent call last):
2023-01-11T21:24:50.2097516Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2023-01-11T21:24:50.2097829Z     self._pc.join(-1)
2023-01-11T21:24:50.2098058Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__
2023-01-11T21:24:50.2098319Z     return _mock_self._mock_call(*args, **kwargs)
2023-01-11T21:24:50.2098585Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call
2023-01-11T21:24:50.2098801Z     raise effect
2023-01-11T21:24:50.2099075Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2023-01-11T21:24:50.2099339Z ok (0.005s)
2023-01-11T21:24:50.2099794Z   test_pcontext_wait (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2100184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2100616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2100965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2101177Z ok (2.683s)
2023-01-11T21:24:50.2101463Z   test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 31503 closing signal SIGTERM
2023-01-11T21:24:50.2101992Z ok (0.010s)
2023-01-11T21:24:50.2102207Z   test_to_map (__main__.StartProcessesListTest) ... ok (0.002s)
2023-01-11T21:24:50.2102501Z   test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s)
2023-01-11T21:24:50.2103051Z   test_void_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2103435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2103858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2104205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2104637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2104962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2105373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2105715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2105934Z world
2023-01-11T21:24:50.2106070Z hello
2023-01-11T21:24:50.2106256Z Closing process 31504 via signal SIGTERM
2023-01-11T21:24:50.2106445Z ok (1.774s)
2023-01-11T21:24:50.2106667Z   test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s)
2023-01-11T21:24:50.2106965Z   test_binary_exit (__main__.StartProcessesTest) ... bar stderr from 1
2023-01-11T21:24:50.2107195Z bar stdout from 1
2023-01-11T21:24:50.2107465Z failed (exitcode: 138) local_rank: 0 (pid: 31524) of binary: distributed/elastic/multiprocessing/bin/echo1.py
2023-01-11T21:24:50.2107726Z ok (0.116s)
2023-01-11T21:24:50.2107967Z   test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.009s)
2023-01-11T21:24:50.2108291Z   test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last):
2023-01-11T21:24:50.2108602Z   File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2023-01-11T21:24:50.2108873Z     raise RuntimeError(f"raised from {rank}")
2023-01-11T21:24:50.2109090Z RuntimeError: raised from 0
2023-01-11T21:24:50.2109260Z bar from 1
2023-01-11T21:24:50.2109534Z failed (exitcode: 1) local_rank: 0 (pid: 31527) of binary: distributed/elastic/multiprocessing/bin/echo2.py
2023-01-11T21:24:50.2109892Z ok (0.113s)
2023-01-11T21:24:50.2110404Z   test_function_large_ret_val (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2110798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2111231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2111575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2111992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2112318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2112742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2113084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2113504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2113832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2114260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2114593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2115021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2115345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2115767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2116097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2116354Z Closing process 31529 via signal SIGTERM
2023-01-11T21:24:50.2116575Z Closing process 31531 via signal SIGTERM
2023-01-11T21:24:50.2116782Z Closing process 31532 via signal SIGTERM
2023-01-11T21:24:50.2116972Z ok (2.070s)
2023-01-11T21:24:50.2117184Z   test_function_raise (__main__.StartProcessesTest)
2023-01-11T21:24:50.2117684Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2118056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2118479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2118818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2119236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2119565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2119990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2120329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2120618Z failed (exitcode: 1) local_rank: 0 (pid: 31569) of fn: echo2 (start_method: spawn)
2023-01-11T21:24:50.2120876Z Traceback (most recent call last):
2023-01-11T21:24:50.2121376Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2023-01-11T21:24:50.2121681Z     self._pc.join(-1)
2023-01-11T21:24:50.2122029Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2023-01-11T21:24:50.2122367Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2023-01-11T21:24:50.2122696Z torch.multiprocessing.spawn.ProcessRaisedException: 
2023-01-11T21:24:50.2122940Z 
2023-01-11T21:24:50.2123103Z -- Process 0 terminated with the following error:
2023-01-11T21:24:50.2123339Z Traceback (most recent call last):
2023-01-11T21:24:50.2123747Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2023-01-11T21:24:50.2123998Z     fn(i, *args)
2023-01-11T21:24:50.2124376Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2023-01-11T21:24:50.2124662Z     ret = record(fn)(*args_)
2023-01-11T21:24:50.2125050Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2023-01-11T21:24:50.2125346Z     return f(*args, **kwargs)
2023-01-11T21:24:50.2125636Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2023-01-11T21:24:50.2125908Z     raise RuntimeError(msg)
2023-01-11T21:24:50.2126085Z RuntimeError: hello
2023-01-11T21:24:50.2126204Z 
2023-01-11T21:24:50.2126270Z ok (1.741s)
2023-01-11T21:24:50.2126504Z   test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.002s)
2023-01-11T21:24:50.2126787Z   test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.001s)
2023-01-11T21:24:50.2127120Z   test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 31589 via signal SIGTERM
2023-01-11T21:24:50.2127378Z ok (0.008s)
2023-01-11T21:24:50.2127815Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2023-01-11T21:24:50.2128157Z Traceback (most recent call last):
2023-01-11T21:24:50.2128555Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2023-01-11T21:24:50.2128864Z     self._pc.join(-1)
2023-01-11T21:24:50.2129089Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__
2023-01-11T21:24:50.2129353Z     return _mock_self._mock_call(*args, **kwargs)
2023-01-11T21:24:50.2129618Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call
2023-01-11T21:24:50.2129834Z     raise effect
2023-01-11T21:24:50.2130111Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2023-01-11T21:24:50.2130371Z ok (0.004s)
2023-01-11T21:24:50.2130814Z   test_pcontext_wait (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2131196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2131622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2131967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2132177Z ok (2.652s)
2023-01-11T21:24:50.2132454Z   test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 31600 closing signal SIGTERM
2023-01-11T21:24:50.2132721Z ok (0.010s)
2023-01-11T21:24:50.2132924Z   test_to_map (__main__.StartProcessesTest) ... ok (0.002s)
2023-01-11T21:24:50.2133202Z   test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s)
2023-01-11T21:24:50.2133729Z   test_void_function (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2134108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2134531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2134876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2135307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:24:50.2135619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:50.2136090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:50.2136461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:50.2136687Z world
2023-01-11T21:24:50.2136826Z hello
2023-01-11T21:24:50.2137009Z Closing process 31602 via signal SIGTERM
2023-01-11T21:24:50.2137203Z ok (1.681s)
2023-01-11T21:24:50.2137402Z   test_from_str_bad_input (__main__.StdTest) ... ok (0.002s)
2023-01-11T21:24:50.2137645Z   test_from_value (__main__.StdTest) ... ok (0.001s)
2023-01-11T21:24:50.2137886Z   test_from_value_map (__main__.StdTest) ... ok (0.001s)
2023-01-11T21:24:50.2138026Z 
2023-01-11T21:24:50.2138217Z ----------------------------------------------------------------------
2023-01-11T21:24:50.2138463Z Ran 38 tests in 20.181s
2023-01-11T21:24:50.2138576Z 
2023-01-11T21:24:50.2138637Z OK
2023-01-11T21:24:50.2138732Z 
2023-01-11T21:24:50.2138820Z Generating XML reports...
2023-01-11T21:24:50.2139299Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20230111212429.xml
2023-01-11T21:24:50.2139975Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20230111212429.xml
2023-01-11T21:24:50.2140633Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20230111212429.xml
2023-01-11T21:24:50.2141232Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20230111212429.xml
2023-01-11T21:24:50.2141496Z 
2023-01-11T21:24:50.2141944Z ##[endgroup]
2023-01-11T21:24:50.2142452Z FINISHED PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_oy3j4zhi)
2023-01-11T21:24:50.2142757Z 
2023-01-11T21:24:50.2142964Z Running distributed/_tensor/test_common_rules ... [2023-01-11 21:24:50.206540]
2023-01-11T21:24:50.2143474Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:24:50.206971]
2023-01-11T21:25:12.8424426Z 
2023-01-11T21:25:12.8425033Z Expand the folded group to see the log file of distributed/_tensor/test_common_rules
2023-01-11T21:25:12.8426154Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_common_rules_se71w651)
2023-01-11T21:25:12.8463243Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_common_rules
2023-01-11T21:25:12.8463735Z 
2023-01-11T21:25:12.8463864Z Running tests...
2023-01-11T21:25:12.8464367Z ----------------------------------------------------------------------
2023-01-11T21:25:12.8465005Z   test_einop_basic_propagation (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31632
2023-01-11T21:25:12.8465681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31633
2023-01-11T21:25:12.8466218Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31634
2023-01-11T21:25:12.8466805Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31635
2023-01-11T21:25:12.8467645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8468098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8468518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8468867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8469191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:12.8469893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8470295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8470725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8471070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8471379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:12.8471838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8472169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8472593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8472922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8473247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:12.8473709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8474024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8474450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8474795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8475117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:12.8475465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:12.8475832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:12.8476204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:12.8476617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:12.8477115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8477633Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8478147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8478643Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8478930Z ok (2.181s)
2023-01-11T21:25:12.8479238Z   test_einop_errors (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31688
2023-01-11T21:25:12.8479616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31689
2023-01-11T21:25:12.8479936Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31690
2023-01-11T21:25:12.8480257Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31691
2023-01-11T21:25:12.8480706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8481024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8481522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8481866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8482189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:12.8482692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8483020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8483480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8483827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8484135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:12.8484592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8484916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8485330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8485671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8485996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:12.8486457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8486768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8487191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8487534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8487845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:12.8488205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:12.8488571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:12.8488935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:12.8489285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:12.8489834Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8490357Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8490875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8491253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T21:25:12.8491614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:25:12.8492103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8492506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:25:12.8492855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:25:12.8493341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8493850Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8494227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:25:12.8494588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T21:25:12.8495073Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8495635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8496055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:25:12.8496425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T21:25:12.8496911Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8497416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8497793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T21:25:12.8498276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8498784Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8499173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T21:25:12.8499525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:25:12.8499883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:25:12.8500372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8500877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8501251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T21:25:12.8501617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T21:25:12.8502274Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8502779Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8503165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T21:25:12.8503527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T21:25:12.8504010Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8504503Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8505011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8505525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8505807Z ok (2.020s)
2023-01-11T21:25:12.8506107Z   test_einop_linearity (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31768
2023-01-11T21:25:12.8506486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31769
2023-01-11T21:25:12.8506817Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31770
2023-01-11T21:25:12.8507140Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31771
2023-01-11T21:25:12.8507582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8507909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8508417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8508792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8509115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:12.8509796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8510125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8510538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8510883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8511205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:12.8511652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8511982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8512407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8512748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8513055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:12.8513512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8513834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8514258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8514587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8514910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:12.8515273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:12.8515626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:12.8516051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:12.8516413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:12.8516910Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8517420Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8517933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8518454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8518842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T21:25:12.8519189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:25:12.8519546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:25:12.8519912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:25:12.8520399Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8520900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8521398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T21:25:12.8521792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:25:12.8522273Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8522784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8523179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:25:12.8523545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T21:25:12.8524013Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8524528Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8525040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8525548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8525922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T21:25:12.8526281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T21:25:12.8526639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:25:12.8526997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:25:12.8527469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8527979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8528364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T21:25:12.8528720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T21:25:12.8529192Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8529700Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8530090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T21:25:12.8530441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T21:25:12.8530923Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8531439Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8531941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8532434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8532712Z ok (1.921s)
2023-01-11T21:25:12.8533030Z   test_einop_merge_sharding (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31848
2023-01-11T21:25:12.8533415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31849
2023-01-11T21:25:12.8533773Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31850
2023-01-11T21:25:12.8534097Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31851
2023-01-11T21:25:12.8534580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8534898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8535324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8535668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8535988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:12.8536436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8536759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8537185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8537528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8537835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:12.8538291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8538616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8539029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8555549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8555895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:12.8556432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8556785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8557233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8557586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8557898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:12.8558266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:12.8558641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:12.8558997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:12.8559363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:12.8559864Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8560389Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8560772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T21:25:12.8561357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8561872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8562267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:25:12.8562619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:25:12.8563153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:25:12.8563716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8564234Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8564612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:25:12.8564979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T21:25:12.8565466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8565969Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8566370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:25:12.8566739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T21:25:12.8567227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8567783Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8568214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T21:25:12.8568753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8569261Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8569644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T21:25:12.8570007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:25:12.8570490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8570893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:25:12.8571362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8571752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T21:25:12.8572110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T21:25:12.8572591Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8573090Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8573481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T21:25:12.8573844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T21:25:12.8574314Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8574821Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8575335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8575841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8576162Z ok (1.921s)
2023-01-11T21:25:12.8576518Z   test_einop_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31928
2023-01-11T21:25:12.8576916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31929
2023-01-11T21:25:12.8577250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31930
2023-01-11T21:25:12.8577568Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31931
2023-01-11T21:25:12.8578018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8578352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8578768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8579118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8579442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:12.8579904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8580221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8580644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8580989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8581311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:12.8581870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8582200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8582637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8582972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8583295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:12.8583749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8584077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8584487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8584830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8585149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:12.8585498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:12.8585866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:12.8586234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:12.8586600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:12.8587079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8587602Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8588117Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8588632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8588971Z ok (1.822s)
2023-01-11T21:25:12.8589336Z   test_einop_pointwise_propagation (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31984
2023-01-11T21:25:12.8589740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31985
2023-01-11T21:25:12.8590064Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31986
2023-01-11T21:25:12.8590400Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31987
2023-01-11T21:25:12.8590850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8591187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8591602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8591956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8592282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:12.8592741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8593057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8593483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8593827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8594132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:12.8594592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8594915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8595343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8595672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8595994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:12.8596452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8596762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8597189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8597531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8597851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:12.8598206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:12.8598574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:12.8598943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:12.8599310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:12.8599788Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8600310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8600823Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8601428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8601743Z ok (1.822s)
2023-01-11T21:25:12.8602123Z   test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32040
2023-01-11T21:25:12.8602546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32041
2023-01-11T21:25:12.8602866Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32042
2023-01-11T21:25:12.8603191Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32043
2023-01-11T21:25:12.8603644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8603974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8604392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8604737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8605060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:12.8605506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8605832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8606261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8606605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8606911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:12.8607376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8607706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8608121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8608459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8608782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:12.8609236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8609553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8609981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8610324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8610646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:12.8611004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:12.8611380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:12.8611747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:12.8612105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:12.8612595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8613114Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8613633Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8614130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8614599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T21:25:12.8614968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:25:12.8615332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:25:12.8615682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:25:12.8616172Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8616683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8617075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T21:25:12.8617428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:25:12.8617922Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8618437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8618827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:25:12.8619182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T21:25:12.8619670Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8620183Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8620565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T21:25:12.8621047Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8621551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8622090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T21:25:12.8622447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:25:12.8622819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:25:12.8623312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8623827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8624210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T21:25:12.8624581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T21:25:12.8625066Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8625580Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8625961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T21:25:12.8626322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T21:25:12.8626812Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8627415Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8627959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8628473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8628764Z ok (1.921s)
2023-01-11T21:25:12.8629080Z   test_pointwise_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32120
2023-01-11T21:25:12.8629477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32121
2023-01-11T21:25:12.8629809Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32122
2023-01-11T21:25:12.8630141Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32123
2023-01-11T21:25:12.8630586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8630922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8631353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8631687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8632010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:12.8632466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8632791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8633203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8633550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8633873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:12.8634328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8634640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8635067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8635412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8635720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:12.8636175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8636500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8636927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8637265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8637585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:12.8637946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:12.8638305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:12.8638669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:12.8639036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:12.8639532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8640079Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8640627Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8641241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8641639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T21:25:12.8641997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:25:12.8642366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:25:12.8642730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:25:12.8643226Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8643607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T21:25:12.8644096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8644484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:25:12.8644970Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8645471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:25:12.8645863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:25:12.8646234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T21:25:12.8646711Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8647223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8647732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8648122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T21:25:12.8648588Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:25:12.8648975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T21:25:12.8649341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:25:12.8649704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:25:12.8650183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8650698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8651092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T21:25:12.8651456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T21:25:12.8651925Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8652433Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:25:12.8652869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T21:25:12.8653271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T21:25:12.8653743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8654261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8654771Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8655261Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:25:12.8655545Z ok (1.920s)
2023-01-11T21:25:12.8655872Z   test_pointwise_rules_broadcasting (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32200
2023-01-11T21:25:12.8656274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32201
2023-01-11T21:25:12.8656589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32202
2023-01-11T21:25:12.8656914Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32203
2023-01-11T21:25:12.8657366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8657693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8658106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8658454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8658779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:12.8659231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8659561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8659985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8660329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8660639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:12.8661092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8661421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8662019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8662373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8662698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:12.8663156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8663471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8663896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8664239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8664559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:12.8664905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:12.8665274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:12.8665753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:12.8666278Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8666678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:12.8667171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8667685Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8668187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8668475Z ok (1.820s)
2023-01-11T21:25:12.8668791Z   test_pointwise_rules_suggestion (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32256
2023-01-11T21:25:12.8669186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32257
2023-01-11T21:25:12.8669504Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32258
2023-01-11T21:25:12.8669836Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32259
2023-01-11T21:25:12.8670286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8670598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8671029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8671375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8671696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:12.8672142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8672469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8672899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8673229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8673547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:12.8674005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8674328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8674739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8675082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8675401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:12.8675858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8676171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8676598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8676942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8677250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:12.8677613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:12.8677986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:12.8678387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:12.8678779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:12.8679273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8679794Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8680306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8680801Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8681158Z ok (1.820s)
2023-01-11T21:25:12.8681477Z   test_reduction_rule (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32312
2023-01-11T21:25:12.8681847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32313
2023-01-11T21:25:12.8682184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32314
2023-01-11T21:25:12.8682508Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32315
2023-01-11T21:25:12.8682962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8683278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8683707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8684051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8684359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:12.8684821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8685151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8685574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8685901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8686220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:12.8686677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8687001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8687411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8687758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8688082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:12.8688526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:12.8688850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:12.8689274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:12.8689619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:12.8689925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:12.8690286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:12.8690655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:12.8691054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:12.8691579Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8691975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:12.8692464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8692968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8693482Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:12.8693771Z ok (1.821s)
2023-01-11T21:25:12.8693885Z 
2023-01-11T21:25:12.8694093Z ----------------------------------------------------------------------
2023-01-11T21:25:12.8694326Z Ran 11 tests in 20.990s
2023-01-11T21:25:12.8694441Z 
2023-01-11T21:25:12.8694503Z OK
2023-01-11T21:25:12.8694595Z 
2023-01-11T21:25:12.8694680Z Generating XML reports...
2023-01-11T21:25:12.8695117Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_common_rules/TEST-CommonRulesTest-20230111212451.xml
2023-01-11T21:25:12.8695367Z 
2023-01-11T21:25:12.8695721Z ##[endgroup]
2023-01-11T21:25:12.8696179Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_common_rules_se71w651)
2023-01-11T21:25:12.8696442Z 
2023-01-11T21:25:12.8696648Z Running distributed/fsdp/test_fsdp_comm_hooks ... [2023-01-11 21:25:12.843400]
2023-01-11T21:25:12.8697161Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:25:12.843765]
2023-01-11T21:25:42.4156928Z 
2023-01-11T21:25:42.4158020Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm_hooks
2023-01-11T21:25:42.4162877Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks__vf50rh4)
2023-01-11T21:25:42.4163966Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks
2023-01-11T21:25:42.4164301Z 
2023-01-11T21:25:42.4164401Z Running tests...
2023-01-11T21:25:42.4164884Z ----------------------------------------------------------------------
2023-01-11T21:25:42.4165538Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:25:42.4166337Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:25:42.4166865Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:25:42.4167382Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:25:42.4167889Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:25:42.4168396Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:25:42.4168924Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4169728Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32379
2023-01-11T21:25:42.4170204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32380
2023-01-11T21:25:42.4170542Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32381
2023-01-11T21:25:42.4170853Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32382
2023-01-11T21:25:42.4171323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4171663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4172103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4172504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4172847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4173322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4173649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4174070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4174421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4174756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4175212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4175540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4175969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4176313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4176633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4177104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4177435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4177850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4178192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4178527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4179024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4179534Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4180061Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4180581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4180972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4181309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4181884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4182288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4182640Z dist init r=0, world=4
2023-01-11T21:25:42.4182813Z dist init r=1, world=4
2023-01-11T21:25:42.4182996Z dist init r=3, world=4
2023-01-11T21:25:42.4183180Z dist init r=2, world=4
2023-01-11T21:25:42.4183430Z skip: Need at least 2 CUDA devices (2.085s)
2023-01-11T21:25:42.4183774Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4184350Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32435
2023-01-11T21:25:42.4184740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32436
2023-01-11T21:25:42.4185072Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32437
2023-01-11T21:25:42.4185413Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32438
2023-01-11T21:25:42.4185870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4186194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4186630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4186983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4187307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4187777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4188109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4188541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4188875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4189212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4189708Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4190207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4190524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4190955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4191296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4191619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4192112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4192611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4192942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4193357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4193703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4194043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4194540Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4195050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4195436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4195833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4196214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4196559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4196820Z dist init r=1, world=4
2023-01-11T21:25:42.4197007Z dist init r=2, world=4
2023-01-11T21:25:42.4197177Z dist init r=3, world=4
2023-01-11T21:25:42.4197357Z dist init r=0, world=4
2023-01-11T21:25:42.4197562Z skip: Need at least 2 CUDA devices (1.822s)
2023-01-11T21:25:42.4197895Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4198475Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32491
2023-01-11T21:25:42.4198889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32492
2023-01-11T21:25:42.4199210Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32493
2023-01-11T21:25:42.4199543Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32494
2023-01-11T21:25:42.4199994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4200326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4200741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4201187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4201533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4202008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4202330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4202766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4203115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4203435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4203905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4204239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4204672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4205000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4205337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4205804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4206119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4206546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4206892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4207228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4207708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4208228Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4208801Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4209348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4209723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4210073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4210426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4210773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4211025Z dist init r=1, world=4
2023-01-11T21:25:42.4211212Z dist init r=3, world=4
2023-01-11T21:25:42.4211391Z dist init r=0, world=4
2023-01-11T21:25:42.4211558Z dist init r=2, world=4
2023-01-11T21:25:42.4211766Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:25:42.4212141Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4212703Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32547
2023-01-11T21:25:42.4213090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32548
2023-01-11T21:25:42.4213417Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32549
2023-01-11T21:25:42.4213743Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32550
2023-01-11T21:25:42.4214185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4214517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4214948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4215288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4215629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4216103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4216433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4216849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4217195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4217530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4218000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4218316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4218741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4219084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4219404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4219905Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4220406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4220734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4221146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4221531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4222062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4222555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4223082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4223612Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4224004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4224341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4224694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4225044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4225312Z dist init r=0, world=4
2023-01-11T21:25:42.4225481Z dist init r=1, world=4
2023-01-11T21:25:42.4225661Z dist init r=3, world=4
2023-01-11T21:25:42.4225837Z dist init r=2, world=4
2023-01-11T21:25:42.4226031Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:25:42.4226632Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4227205Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32603
2023-01-11T21:25:42.4227597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32604
2023-01-11T21:25:42.4227915Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32605
2023-01-11T21:25:42.4228247Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32606
2023-01-11T21:25:42.4228703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4229020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4229448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4229794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4230131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4230586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4230913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4231340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4231674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4232006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4232474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4232798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4233211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4233561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4233896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4234365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4234732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4235205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4235549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4235870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4236367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4236893Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4237416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4237920Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4238312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4238662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4239010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4239346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4239600Z dist init r=0, world=4
2023-01-11T21:25:42.4239783Z dist init r=1, world=4
2023-01-11T21:25:42.4239948Z dist init r=2, world=4
2023-01-11T21:25:42.4240125Z dist init r=3, world=4
2023-01-11T21:25:42.4240333Z skip: Need at least 2 CUDA devices (1.823s)
2023-01-11T21:25:42.4240693Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4241337Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32659
2023-01-11T21:25:42.4241734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32660
2023-01-11T21:25:42.4242076Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32661
2023-01-11T21:25:42.4242390Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32662
2023-01-11T21:25:42.4242848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4243189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4243627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4243965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4244305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4244779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4245096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4245527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4245874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4246206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4246662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4246985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4247409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4247798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4248181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4248686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4249187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4249503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4249931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4250276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4250615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4251104Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4251628Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4252149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4252530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4252866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4253204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4253547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4253798Z dist init r=3, world=4
2023-01-11T21:25:42.4253980Z dist init r=0, world=4
2023-01-11T21:25:42.4254161Z dist init r=1, world=4
2023-01-11T21:25:42.4254326Z dist init r=2, world=4
2023-01-11T21:25:42.4254535Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:25:42.4254904Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4255464Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32715
2023-01-11T21:25:42.4255835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32716
2023-01-11T21:25:42.4256164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32717
2023-01-11T21:25:42.4256493Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32718
2023-01-11T21:25:42.4256929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4257264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4257699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4258043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4258363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4258829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4259157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4259571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4259914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4260289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4260796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4261111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4261540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4262105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4262445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4262909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4263246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4263679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4264013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4264354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4264854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4265377Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4265883Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4266401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4266791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4267147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4267483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4267828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4268088Z dist init r=2, world=4
2023-01-11T21:25:42.4268259Z dist init r=3, world=4
2023-01-11T21:25:42.4268441Z dist init r=0, world=4
2023-01-11T21:25:42.4268621Z dist init r=1, world=4
2023-01-11T21:25:42.4268816Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:25:42.4269187Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4269754Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 303
2023-01-11T21:25:42.4270135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 304
2023-01-11T21:25:42.4270451Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 305
2023-01-11T21:25:42.4270775Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 306
2023-01-11T21:25:42.4271226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4271557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4271975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4272320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4272655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4273112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4273531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4274010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4274364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4274685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4275162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4275497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4275910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4276254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4276591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4277064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4277377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4277810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4278158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4278496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4278979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4279497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4280015Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4280531Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4280904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4281338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4281689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4282023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4282281Z dist init r=3, world=4
2023-01-11T21:25:42.4282470Z dist init r=0, world=4
2023-01-11T21:25:42.4282639Z dist init r=1, world=4
2023-01-11T21:25:42.4282822Z dist init r=2, world=4
2023-01-11T21:25:42.4283031Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:25:42.4283410Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4283967Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 359
2023-01-11T21:25:42.4284350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 360
2023-01-11T21:25:42.4284673Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 361
2023-01-11T21:25:42.4284982Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 362
2023-01-11T21:25:42.4285437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4285772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4286199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4286599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4286969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4287439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4287765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4288179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4288520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4288858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4289310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4289647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4290077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4290421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4290740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4291207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4291535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4291946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4292287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4292621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4293121Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4293628Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4294142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4294661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4295045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4295379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4295715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4296063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4296307Z dist init r=1, world=4
2023-01-11T21:25:42.4296492Z dist init r=3, world=4
2023-01-11T21:25:42.4296670Z dist init r=0, world=4
2023-01-11T21:25:42.4296833Z dist init r=2, world=4
2023-01-11T21:25:42.4297039Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:25:42.4297431Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:25:42.4297950Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:25:42.4298449Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:25:42.4299047Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:25:42.4299535Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:25:42.4300052Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:25:42.4300511Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4301047Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 415
2023-01-11T21:25:42.4301428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 416
2023-01-11T21:25:42.4303081Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 417
2023-01-11T21:25:42.4303412Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 418
2023-01-11T21:25:42.4303880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4304201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4304631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4304983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4305323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4305786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4306121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4306552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4306884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4307221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4307721Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4308228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4308545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4308976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4309331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4309652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4310121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4310450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4310876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4311206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4311544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4312044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4312653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4313204Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4313594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4313945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4314293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4314624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4314879Z dist init r=3, world=4
2023-01-11T21:25:42.4315060Z dist init r=0, world=4
2023-01-11T21:25:42.4315228Z dist init r=2, world=4
2023-01-11T21:25:42.4315407Z dist init r=1, world=4
2023-01-11T21:25:42.4315611Z skip: Need at least 2 CUDA devices (2.020s)
2023-01-11T21:25:42.4315926Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4316465Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 471
2023-01-11T21:25:42.4316856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 472
2023-01-11T21:25:42.4317174Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 473
2023-01-11T21:25:42.4317484Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 474
2023-01-11T21:25:42.4317929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4318258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4318674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4319026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4319364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4319833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4320147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4320572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4320918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4321329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4321800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4322133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4322559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4322892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4323229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4323697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4324021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4324438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4324779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4325110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4325648Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4326208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4326733Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4327251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4327623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4327969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4328309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4328657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4328902Z dist init r=3, world=4
2023-01-11T21:25:42.4329084Z dist init r=1, world=4
2023-01-11T21:25:42.4329267Z dist init r=0, world=4
2023-01-11T21:25:42.4329431Z dist init r=2, world=4
2023-01-11T21:25:42.4329633Z skip: Need at least 2 CUDA devices (1.822s)
2023-01-11T21:25:42.4329969Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4330495Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 527
2023-01-11T21:25:42.4330882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 528
2023-01-11T21:25:42.4331209Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 529
2023-01-11T21:25:42.4331526Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 530
2023-01-11T21:25:42.4331963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4332294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4332722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4333053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4333387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4333853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4334180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4334592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4334940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4335269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4335737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4336048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4336470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4336815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4337132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4337594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4337918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4338382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4338764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4339101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4339605Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4340124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4340625Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4341139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4341527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4342082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4342431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4342774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4343036Z dist init r=0, world=4
2023-01-11T21:25:42.4343208Z dist init r=2, world=4
2023-01-11T21:25:42.4343389Z dist init r=3, world=4
2023-01-11T21:25:42.4343569Z dist init r=1, world=4
2023-01-11T21:25:42.4343764Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:25:42.4344100Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4344648Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 583
2023-01-11T21:25:42.4345039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 584
2023-01-11T21:25:42.4345351Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 585
2023-01-11T21:25:42.4345666Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 586
2023-01-11T21:25:42.4346117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4346435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4346861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4347204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4347540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4348000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4348325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4348754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4349083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4349416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4349889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4350215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4350626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4350969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4351396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4351918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4352236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4352667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4353011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4353331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4353824Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4354343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4354864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4355363Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4355746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4356091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4356436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4356771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4357027Z dist init r=2, world=4
2023-01-11T21:25:42.4357210Z dist init r=0, world=4
2023-01-11T21:25:42.4357376Z dist init r=3, world=4
2023-01-11T21:25:42.4357559Z dist init r=1, world=4
2023-01-11T21:25:42.4357771Z skip: Need at least 2 CUDA devices (1.822s)
2023-01-11T21:25:42.4358100Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4358649Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 639
2023-01-11T21:25:42.4359038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 641
2023-01-11T21:25:42.4359364Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 642
2023-01-11T21:25:42.4359674Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 643
2023-01-11T21:25:42.4360131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4360467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4360887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4361315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4361660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4362139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4362459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4362890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4363237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4363574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4364029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4364407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4364867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4365203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4365539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4366008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4366333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4366745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4367094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4367435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4367920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4368438Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4368955Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4369474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4369845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4370194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4370537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4370880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4371129Z dist init r=3, world=4
2023-01-11T21:25:42.4371311Z dist init r=1, world=4
2023-01-11T21:25:42.4371489Z dist init r=0, world=4
2023-01-11T21:25:42.4371651Z dist init r=2, world=4
2023-01-11T21:25:42.4371857Z skip: Need at least 2 CUDA devices (1.822s)
2023-01-11T21:25:42.4372192Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T21:25:42.4372722Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 697
2023-01-11T21:25:42.4373109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 698
2023-01-11T21:25:42.4373431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 699
2023-01-11T21:25:42.4373753Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 700
2023-01-11T21:25:42.4374188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4374518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4374948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4375298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4375621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:25:42.4376092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4376417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4376830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4377214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4377581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:25:42.4378048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4378361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4378785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4379128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4379451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:25:42.4379912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:25:42.4380238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:25:42.4380662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:25:42.4380991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:25:42.4381322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:25:42.4382065Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4382595Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4383105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4383623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:25:42.4384016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:25:42.4384369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:25:42.4384704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:25:42.4385046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:25:42.4385308Z dist init r=3, world=4
2023-01-11T21:25:42.4385478Z dist init r=2, world=4
2023-01-11T21:25:42.4385655Z dist init r=0, world=4
2023-01-11T21:25:42.4385833Z dist init r=1, world=4
2023-01-11T21:25:42.4386026Z skip: Need at least 2 CUDA devices (1.920s)
2023-01-11T21:25:42.4386168Z 
2023-01-11T21:25:42.4386371Z ----------------------------------------------------------------------
2023-01-11T21:25:42.4386614Z Ran 27 tests in 27.889s
2023-01-11T21:25:42.4386727Z 
2023-01-11T21:25:42.4386800Z OK (skipped=27)
2023-01-11T21:25:42.4386897Z 
2023-01-11T21:25:42.4386982Z Generating XML reports...
2023-01-11T21:25:42.4387456Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20230111212514.xml
2023-01-11T21:25:42.4387732Z 
2023-01-11T21:25:42.4388100Z ##[endgroup]
2023-01-11T21:25:42.4388552Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks__vf50rh4)
2023-01-11T21:25:42.4388818Z 
2023-01-11T21:25:42.4389012Z Running distributed/_tensor/test_dtensor ... [2023-01-11 21:25:42.416755]
2023-01-11T21:25:42.4389528Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:25:42.417114]
2023-01-11T21:26:11.8899125Z 
2023-01-11T21:26:11.8900166Z Expand the folded group to see the log file of distributed/_tensor/test_dtensor
2023-01-11T21:26:11.8901632Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_dtensor_0vd_c568)
2023-01-11T21:26:11.8902698Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_dtensor
2023-01-11T21:26:11.8903034Z 
2023-01-11T21:26:11.8903160Z Running tests...
2023-01-11T21:26:11.8903679Z ----------------------------------------------------------------------
2023-01-11T21:26:11.8904461Z   test_device_mesh_nd (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 764
2023-01-11T21:26:11.8905065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 765
2023-01-11T21:26:11.8905573Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 766
2023-01-11T21:26:11.8906099Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 767
2023-01-11T21:26:11.8906640Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 768
2023-01-11T21:26:11.8907188Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 769
2023-01-11T21:26:11.8907745Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 770
2023-01-11T21:26:11.8908289Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 771
2023-01-11T21:26:11.8909183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.8909833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.8910596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.8911160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.8911724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:26:11.8912609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.8913209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.8914009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.8914603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.8915185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:11.8916051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.8916595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.8917352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.8917987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.8918593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:11.8919482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.8920038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.8920837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.8921461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.8921945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:26:11.8922692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.8923219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.8924086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.8924727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.8925260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:26:11.8926023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.8926547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.8927235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.8927766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.8928286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:26:11.8929094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.8929643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.8930370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.8930939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.8931519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:26:11.8932376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.8932968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.8933750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.8934358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.8934956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:26:11.8935601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:11.8936236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:11.8936823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:26:11.8937432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:26:11.8938036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T21:26:11.8938613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T21:26:11.8939179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T21:26:11.8939786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T21:26:11.8940595Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.8941426Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.8942407Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.8943208Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.8943978Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.8944572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T21:26:11.8945280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T21:26:11.8945952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T21:26:11.8946513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T21:26:11.8947348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.8948189Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.8948967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.8949395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:26:11.8949767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:26:11.8950133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:26:11.8950483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T21:26:11.8950983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.8951497Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.8951888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:26:11.8952240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T21:26:11.8952727Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.8954753Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.8955153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T21:26:11.8955632Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.8956024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T21:26:11.8956507Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.8956900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T21:26:11.8957377Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.8957888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.8958284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:26:11.8958645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T21:26:11.8959122Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.8959512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T21:26:11.8959993Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.8960381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T21:26:11.8960815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T21:26:11.8961439Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.8961957Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.8962341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T21:26:11.8962828Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.8963219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T21:26:11.8963697Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.8964199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.8964712Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.8965101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:26:11.8965462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T21:26:11.8965812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:26:11.8966173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T21:26:11.8966651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.8967159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.8967660Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.8968169Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.8968557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T21:26:11.8968915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T21:26:11.8969259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T21:26:11.8969612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T21:26:11.8970093Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.8970592Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.8971098Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.8971610Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.8971999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T21:26:11.8972346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T21:26:11.8972706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T21:26:11.8973064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T21:26:11.8973585Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.8974116Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.8974507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2023-01-11T21:26:11.8974864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T21:26:11.8975349Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.8975848Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.8976356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.8976861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.8977247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2023-01-11T21:26:11.8977715Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.8978222Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.8978607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T21:26:11.8978953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2023-01-11T21:26:11.8979309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2023-01-11T21:26:11.8979665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T21:26:11.8980018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T21:26:11.8980494Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.8981002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.8981515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.8982178Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.8982556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2023-01-11T21:26:11.8983039Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.8983428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T21:26:11.8983791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2023-01-11T21:26:11.8984137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2023-01-11T21:26:11.8984626Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.8985133Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.8985520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2023-01-11T21:26:11.8985992Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.8986451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2023-01-11T21:26:11.8986849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T21:26:11.8987198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2023-01-11T21:26:11.8987685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.8988191Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.8988580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2023-01-11T21:26:11.8988927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3
2023-01-11T21:26:11.8989407Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.8989924Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.8990430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.8990808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2
2023-01-11T21:26:11.8991285Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.8991791Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.8992178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2023-01-11T21:26:11.8992524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5
2023-01-11T21:26:11.8993011Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.8993401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7
2023-01-11T21:26:11.8993763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4
2023-01-11T21:26:11.8994104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6
2023-01-11T21:26:11.8994587Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T21:26:11.8995099Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T21:26:11.8995475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4
2023-01-11T21:26:11.8995838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6
2023-01-11T21:26:11.8996326Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T21:26:11.8996836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T21:26:11.8997333Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T21:26:11.8997716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2023-01-11T21:26:11.8998077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3
2023-01-11T21:26:11.8998558Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T21:26:11.8999093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T21:26:11.8999511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2
2023-01-11T21:26:11.8999996Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T21:26:11.9000384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2023-01-11T21:26:11.9000732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5
2023-01-11T21:26:11.9001161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7
2023-01-11T21:26:11.9001651Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T21:26:11.9002166Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T21:26:11.9002668Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T21:26:11.9003058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5
2023-01-11T21:26:11.9003424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6
2023-01-11T21:26:11.9003779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7
2023-01-11T21:26:11.9004264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T21:26:11.9004777Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T21:26:11.9005167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3
2023-01-11T21:26:11.9005639Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T21:26:11.9006149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T21:26:11.9006542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2
2023-01-11T21:26:11.9007023Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T21:26:11.9007395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4
2023-01-11T21:26:11.9007758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0
2023-01-11T21:26:11.9008130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1
2023-01-11T21:26:11.9008622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T21:26:11.9009125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T21:26:11.9009514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1
2023-01-11T21:26:11.9009878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0
2023-01-11T21:26:11.9010367Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T21:26:11.9010867Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T21:26:11.9011369Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T21:26:11.9011920Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T21:26:11.9012462Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T21:26:11.9012843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5
2023-01-11T21:26:11.9013210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6
2023-01-11T21:26:11.9013572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7
2023-01-11T21:26:11.9013924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4
2023-01-11T21:26:11.9014410Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T21:26:11.9014798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2
2023-01-11T21:26:11.9015162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3
2023-01-11T21:26:11.9015634Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T21:26:11.9016147Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T21:26:11.9016539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2
2023-01-11T21:26:11.9016903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3
2023-01-11T21:26:11.9017376Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T21:26:11.9017881Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T21:26:11.9018391Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T21:26:11.9018900Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T21:26:11.9019279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6
2023-01-11T21:26:11.9019644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7
2023-01-11T21:26:11.9020129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T21:26:11.9020633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T21:26:11.9021007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1
2023-01-11T21:26:11.9021369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0
2023-01-11T21:26:11.9036871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4
2023-01-11T21:26:11.9037255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5
2023-01-11T21:26:11.9037801Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T21:26:11.9038322Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T21:26:11.9038722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5
2023-01-11T21:26:11.9039093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4
2023-01-11T21:26:11.9039703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T21:26:11.9040276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T21:26:11.9040672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1
2023-01-11T21:26:11.9041124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0
2023-01-11T21:26:11.9041610Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T21:26:11.9042128Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T21:26:11.9042521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2
2023-01-11T21:26:11.9042889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3
2023-01-11T21:26:11.9043371Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T21:26:11.9043887Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T21:26:11.9044287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6
2023-01-11T21:26:11.9044658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7
2023-01-11T21:26:11.9045137Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T21:26:11.9045651Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T21:26:11.9046173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T21:26:11.9046692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T21:26:11.9047195Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T21:26:11.9047705Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T21:26:11.9048216Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T21:26:11.9049012Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T21:26:11.9049288Z ok (3.403s)
2023-01-11T21:26:11.9049600Z   test_dtensor_2d_mesh (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 948
2023-01-11T21:26:11.9049978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 949
2023-01-11T21:26:11.9050307Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 950
2023-01-11T21:26:11.9050622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 951
2023-01-11T21:26:11.9050943Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 952
2023-01-11T21:26:11.9051265Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 953
2023-01-11T21:26:11.9051576Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 954
2023-01-11T21:26:11.9051894Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 955
2023-01-11T21:26:11.9052348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9052686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9053154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9053536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9053868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:11.9054320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9054649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9055077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9055422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9055731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:26:11.9056200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9056527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9056941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9057281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9057607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:26:11.9058068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9058381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9058805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9059150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9059480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:26:11.9059930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9060257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9060681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9061013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9061335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:26:11.9061953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9062289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9062709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9063054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9063376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:26:11.9063822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9064153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9064581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9064931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9065238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:11.9065695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9066088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9066576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9066911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9067235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:26:11.9067599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:11.9067958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:11.9068327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:26:11.9068696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:26:11.9069066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T21:26:11.9069426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T21:26:11.9069789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T21:26:11.9070283Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9070684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T21:26:11.9071157Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9071671Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9072068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T21:26:11.9072425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T21:26:11.9072790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T21:26:11.9073276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9073790Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9074294Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9074680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:26:11.9075046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:26:11.9075412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T21:26:11.9075889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9076408Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9076801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:26:11.9077165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T21:26:11.9077642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9078157Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9078593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:26:11.9078977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T21:26:11.9079465Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9079858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T21:26:11.9080342Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9080841Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9081314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T21:26:11.9081807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9082319Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9082819Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9083210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T21:26:11.9083575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T21:26:11.9083942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:26:11.9084294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T21:26:11.9084779Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9085298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9085689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T21:26:11.9086040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:26:11.9086531Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9087044Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9087437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T21:26:11.9087908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9088416Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9088808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:26:11.9089160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T21:26:11.9089647Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9090161Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9090554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T21:26:11.9090904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T21:26:11.9091306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T21:26:11.9091819Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9092329Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9092827Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9093215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T21:26:11.9093698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9094092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T21:26:11.9094448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T21:26:11.9094818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T21:26:11.9095304Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9095814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9096312Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9096705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T21:26:11.9097068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T21:26:11.9097540Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9097930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T21:26:11.9098295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T21:26:11.9098782Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9099273Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9099663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2023-01-11T21:26:11.9100146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9100658Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9101042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2023-01-11T21:26:11.9101527Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9102177Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9102680Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9103173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9103567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2023-01-11T21:26:11.9103931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2023-01-11T21:26:11.9104358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T21:26:11.9104751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T21:26:11.9105244Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9105634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T21:26:11.9105986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T21:26:11.9106469Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9106980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9107377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2023-01-11T21:26:11.9107729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2023-01-11T21:26:11.9108091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T21:26:11.9108577Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9109090Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9109584Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9110091Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9110603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9110996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T21:26:11.9111350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2023-01-11T21:26:11.9111715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2023-01-11T21:26:11.9112076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2023-01-11T21:26:11.9112437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2023-01-11T21:26:11.9112912Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9113424Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9113935Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9114428Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9114936Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9115452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9115961Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9116454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9116739Z ok (3.040s)
2023-01-11T21:26:11.9117113Z   test_dtensor_api_device_mesh_context_manager (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1108
2023-01-11T21:26:11.9117548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1109
2023-01-11T21:26:11.9117870Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1110
2023-01-11T21:26:11.9118196Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1111
2023-01-11T21:26:11.9118522Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1112
2023-01-11T21:26:11.9118835Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1113
2023-01-11T21:26:11.9119165Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1114
2023-01-11T21:26:11.9119489Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1115
2023-01-11T21:26:11.9119941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9120265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9120699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9121125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9121457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:26:11.9121909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9122242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9122671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9123006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9123336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:26:11.9123799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9124129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9124538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9124886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9125209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:26:11.9125655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9125984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9126408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9126757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9127071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:26:11.9127528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9127854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9128283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9128618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9128939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:11.9129399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9129752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9130212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9130555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9130879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:26:11.9131324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9131654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9132082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9132417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9132746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:11.9133212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9133541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9133951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9134300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9134625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:26:11.9134989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:11.9135349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:11.9135717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:26:11.9136091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:26:11.9136448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T21:26:11.9136814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T21:26:11.9137179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T21:26:11.9137539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T21:26:11.9138024Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9138545Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9139062Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9139580Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9140083Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9140600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9141115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9141624Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9142044Z ok (2.741s)
2023-01-11T21:26:11.9142384Z   test_dtensor_device_mesh_device_conversion (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1220
2023-01-11T21:26:11.9142855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1221
2023-01-11T21:26:11.9143213Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1222
2023-01-11T21:26:11.9143545Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1223
2023-01-11T21:26:11.9143870Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1224
2023-01-11T21:26:11.9144202Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1225
2023-01-11T21:26:11.9144514Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1226
2023-01-11T21:26:11.9144834Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1227
2023-01-11T21:26:11.9145293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9145618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9146053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9146405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9146729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:26:11.9147175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9147509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9147941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9148285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9148595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:26:11.9149057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9149388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9149805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9150149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9150469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:26:11.9150926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9151239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9151665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9152015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9152323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:11.9152788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9153115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9153542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9153875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9154197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:26:11.9154656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9154984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9155433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9155833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9156156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:11.9156604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9156930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9157359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9157704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9158015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:26:11.9158473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9158799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9159217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9159561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9159883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:26:11.9160247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:11.9160604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:11.9160973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:26:11.9161413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:26:11.9161790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T21:26:11.9162144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T21:26:11.9162506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T21:26:11.9162872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T21:26:11.9163352Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9163875Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9164391Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9164909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9165411Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9165922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9166432Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9166945Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9167220Z ok (2.739s)
2023-01-11T21:26:11.9167547Z   test_dtensor_spec_local_shard_offset (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1332
2023-01-11T21:26:11.9168011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1333
2023-01-11T21:26:11.9168340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1334
2023-01-11T21:26:11.9168686Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1335
2023-01-11T21:26:11.9169014Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1336
2023-01-11T21:26:11.9169344Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1337
2023-01-11T21:26:11.9169656Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1338
2023-01-11T21:26:11.9169977Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1339
2023-01-11T21:26:11.9170427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9170759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9171175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9171531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9171857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:26:11.9172303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9172637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9173072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9173422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9173729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:26:11.9174187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9174520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9174941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9175286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9175608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:26:11.9176073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9176387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9176814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9177164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9177490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:11.9177941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9178268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9178694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9179030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9179351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:11.9179812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9180141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9180555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9180941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9181293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:26:11.9181874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9182210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9182640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9182987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9183294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:26:11.9183754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9184085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9184518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9184848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9185174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:26:11.9185538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:11.9185895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:11.9186263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:26:11.9186634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:26:11.9187002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T21:26:11.9187356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T21:26:11.9187719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T21:26:11.9188208Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9188606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T21:26:11.9189078Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9189470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T21:26:11.9189834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T21:26:11.9190308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9190825Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9191341Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9191854Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9192348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9192856Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T21:26:11.9193249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T21:26:11.9193679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:26:11.9194066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:26:11.9194439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T21:26:11.9194804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:26:11.9195166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T21:26:11.9195646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9196165Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9196558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:26:11.9196923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T21:26:11.9197395Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9197908Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9198299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T21:26:11.9198773Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9199287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9199673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T21:26:11.9200156Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9200657Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T21:26:11.9201136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T21:26:11.9201506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T21:26:11.9201873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:26:11.9202223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T21:26:11.9202713Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9203230Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9203745Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9204124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T21:26:11.9204493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T21:26:11.9204981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9205373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:26:11.9205840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9206391Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9206812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:26:11.9207178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T21:26:11.9207657Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9208169Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T21:26:11.9208562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T21:26:11.9208918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T21:26:11.9209281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T21:26:11.9209771Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9210284Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9210783Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9211173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T21:26:11.9211656Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9212043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T21:26:11.9212513Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9212905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T21:26:11.9213387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9213776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T21:26:11.9214246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9214759Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T21:26:11.9215153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T21:26:11.9215517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T21:26:11.9215871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T21:26:11.9216235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T21:26:11.9216724Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9217224Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9217737Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9218248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9218637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2023-01-11T21:26:11.9219021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2023-01-11T21:26:11.9219533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9220048Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9220456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2023-01-11T21:26:11.9220930Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9221437Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T21:26:11.9221965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2023-01-11T21:26:11.9222339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T21:26:11.9222691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T21:26:11.9223059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T21:26:11.9223424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T21:26:11.9223896Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9224410Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9224804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2023-01-11T21:26:11.9225169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2023-01-11T21:26:11.9225645Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9226159Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9226669Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9227178Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9227674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9228183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T21:26:11.9228572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T21:26:11.9228938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T21:26:11.9229292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2023-01-11T21:26:11.9229656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2023-01-11T21:26:11.9230014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2023-01-11T21:26:11.9230376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2023-01-11T21:26:11.9230852Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9231372Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9231944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9232492Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9232991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9233512Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9234023Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9234533Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T21:26:11.9234805Z ok (3.040s)
2023-01-11T21:26:11.9235121Z   test_dtensor_constructor (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1492
2023-01-11T21:26:11.9235504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1493
2023-01-11T21:26:11.9235825Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1494
2023-01-11T21:26:11.9236149Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1495
2023-01-11T21:26:11.9236606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9236941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9237358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9237708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9238033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:26:11.9238483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9238823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9239256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9239604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9239916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:11.9240374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9240708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9241183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9241531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9241857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:11.9242320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9242635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9243064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9243413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9243736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:26:11.9244090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:11.9244459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:26:11.9244992Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9245419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:11.9245791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:26:11.9246283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9246802Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9247302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9247587Z ok (1.820s)
2023-01-11T21:26:11.9247900Z   test_dtensor_properties (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1548
2023-01-11T21:26:11.9248284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1549
2023-01-11T21:26:11.9248601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1550
2023-01-11T21:26:11.9248924Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1551
2023-01-11T21:26:11.9249376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9249693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9250122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9250471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9250798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:11.9251250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9251577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9252007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9252340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9252663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:11.9253127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9253455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9253871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9254214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9254538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:26:11.9255003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9255319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9255744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9256091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9256405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:26:11.9256769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:11.9257139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:11.9257543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:26:11.9257924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:26:11.9258421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9258943Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9259460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9259960Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9260248Z ok (1.820s)
2023-01-11T21:26:11.9260575Z   test_dtensor_spec_read_only_after_set (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1604
2023-01-11T21:26:11.9260959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1605
2023-01-11T21:26:11.9261294Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1606
2023-01-11T21:26:11.9261622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1607
2023-01-11T21:26:11.9262209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9262532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9262958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9263308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9263621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:26:11.9264090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9264419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9264851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9265187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9265509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:26:11.9265967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9266295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9266709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9267055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9267381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:11.9267830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9268156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9268582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9268926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9269234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:11.9269602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:11.9269976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:11.9270400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:26:11.9270785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:26:11.9271281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9271799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9272297Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9272813Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9273107Z ok (1.819s)
2023-01-11T21:26:11.9273410Z   test_dtensor_stride (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1660
2023-01-11T21:26:11.9273772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1661
2023-01-11T21:26:11.9274105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1662
2023-01-11T21:26:11.9274437Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1663
2023-01-11T21:26:11.9274895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9275218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9275650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9275997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9276309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:11.9276769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9277096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9277523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9277854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9278180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:11.9278642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9278956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9279385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9279730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9280057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:26:11.9280503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9280832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9281329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9281683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9281995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:26:11.9282359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:11.9282729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:11.9283126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:26:11.9283525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:26:11.9284023Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9284542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9285048Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9285558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9285852Z ok (1.822s)
2023-01-11T21:26:11.9286148Z   test_from_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1716
2023-01-11T21:26:11.9286507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1717
2023-01-11T21:26:11.9286840Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1718
2023-01-11T21:26:11.9287170Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1719
2023-01-11T21:26:11.9287607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9287938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9288365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9288713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9289025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:11.9289490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9289819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9290236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9290585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9290909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:26:11.9291368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9291680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9292115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9292463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9292776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:26:11.9293238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9293568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9293996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9294329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9294657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:11.9295063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:11.9295474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:11.9295832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:26:11.9296242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:26:11.9296767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9297270Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9297784Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9298308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9298596Z ok (1.821s)
2023-01-11T21:26:11.9298896Z   test_from_local_then_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1772
2023-01-11T21:26:11.9299281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1773
2023-01-11T21:26:11.9299613Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1774
2023-01-11T21:26:11.9299944Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1775
2023-01-11T21:26:11.9300385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9300722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9301191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9301529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9301978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:11.9302442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9302777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9303198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9303551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9303878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:11.9304336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9304654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9305084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9305431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9305742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:26:11.9306207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9306535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9306966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9307299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9307619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:26:11.9307984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:11.9308343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:26:11.9308842Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9309340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:11.9309707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:26:11.9310187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9310715Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9311232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9311519Z ok (1.920s)
2023-01-11T21:26:11.9311800Z   test_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1828
2023-01-11T21:26:11.9312172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1829
2023-01-11T21:26:11.9312505Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1830
2023-01-11T21:26:11.9312826Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1831
2023-01-11T21:26:11.9313277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9313608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9314036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9314368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9314692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:11.9315154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9315483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9315901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9316251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9316577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:11.9317023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9317350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9317784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9318131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9318439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:26:11.9318899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:11.9319221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:11.9319638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:11.9319980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:11.9320300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:26:11.9320670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:11.9321122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:11.9321494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:26:11.9321898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:26:11.9322425Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9322934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9323457Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9323978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:26:11.9324254Z ok (1.820s)
2023-01-11T21:26:11.9324362Z 
2023-01-11T21:26:11.9324564Z ----------------------------------------------------------------------
2023-01-11T21:26:11.9324812Z Ran 12 tests in 27.807s
2023-01-11T21:26:11.9324931Z 
2023-01-11T21:26:11.9324994Z OK
2023-01-11T21:26:11.9325074Z 
2023-01-11T21:26:11.9325158Z Generating XML reports...
2023-01-11T21:26:11.9325597Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20230111212543.xml
2023-01-11T21:26:11.9326148Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20230111212543.xml
2023-01-11T21:26:11.9326387Z 
2023-01-11T21:26:11.9326719Z ##[endgroup]
2023-01-11T21:26:11.9327165Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_dtensor_0vd_c568)
2023-01-11T21:26:11.9327421Z 
2023-01-11T21:26:11.9327615Z Running distributed/fsdp/test_fsdp_misc ... [2023-01-11 21:26:11.892415]
2023-01-11T21:26:11.9328125Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:26:11.892814]
2023-01-11T21:26:44.2907215Z 
2023-01-11T21:26:44.2908199Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_misc
2023-01-11T21:26:44.2909463Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_misc_yo36j1on)
2023-01-11T21:26:44.2910530Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_misc
2023-01-11T21:26:44.2910910Z 
2023-01-11T21:26:44.2912717Z Running tests...
2023-01-11T21:26:44.2913290Z ----------------------------------------------------------------------
2023-01-11T21:26:44.2913822Z   test_cpu_init_with_sync_module_states (__main__.TestFSDPMisc)
2023-01-11T21:26:44.2914438Z Tests that passing ``sync_module_states=True`` raises an error for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1895
2023-01-11T21:26:44.2915073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1896
2023-01-11T21:26:44.2915899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2916531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2917041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2917387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2917709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.2918180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2918505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2918913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2919486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2919825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.2920584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2921241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2921635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.2922062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.2922326Z dist init r=1, world=2
2023-01-11T21:26:44.2922495Z dist init r=0, world=2
2023-01-11T21:26:44.2922739Z skip: Need at least 2 CUDA devices (1.974s)
2023-01-11T21:26:44.2923016Z   test_device_id_auto_wrap (__main__.TestFSDPMisc)
2023-01-11T21:26:44.2923368Z Tests that ``auto_wrap_policy`` propagates ``device_id`` to all ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1923
2023-01-11T21:26:44.2923830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1924
2023-01-11T21:26:44.2924312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2924667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2925134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2925501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2925844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.2926357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2926737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2927182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2927592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2927916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.2928479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2929006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2929443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.2929790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.2930115Z dist init r=1, world=2
2023-01-11T21:26:44.2930297Z dist init r=0, world=2
2023-01-11T21:26:44.2930492Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:26:44.2930811Z   test_fsdp_cpu_init_stays_on_cpu (__main__.TestFSDPMisc)
2023-01-11T21:26:44.2931190Z Tests that passing a CPU module to FSDP preserves that the wrapped ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1951
2023-01-11T21:26:44.2931631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1952
2023-01-11T21:26:44.2932089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2932485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2932913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2933317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2933727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.2934317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2934703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2935136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2935545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2935881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.2936426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2936964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2937395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.2937766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.2938059Z dist init r=1, world=2
2023-01-11T21:26:44.2938242Z dist init r=0, world=2
2023-01-11T21:26:44.2938453Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:26:44.2938757Z   test_fsdp_device_id_cpu_offload (__main__.TestFSDPMisc)
2023-01-11T21:26:44.2939125Z Ensures that even if device_id is specified but we have ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1979
2023-01-11T21:26:44.2939569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1980
2023-01-11T21:26:44.2940012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2940411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2940846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2941265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2941592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.2942277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2942663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2943095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2943477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2943819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.2944373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2944889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2945280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.2945676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.2945937Z dist init r=0, world=2
2023-01-11T21:26:44.2946105Z dist init r=1, world=2
2023-01-11T21:26:44.2946313Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:26:44.2946572Z   test_fsdp_device_id_use_index_False (__main__.TestFSDPMisc)
2023-01-11T21:26:44.2946908Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2007
2023-01-11T21:26:44.2947282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2008
2023-01-11T21:26:44.2947881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2948270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2948691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2949032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2949363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.2949818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2950147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2950571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2950916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2951240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.2951796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2952315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2952700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.2953032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.2953283Z dist init r=1, world=2
2023-01-11T21:26:44.2953462Z dist init r=0, world=2
2023-01-11T21:26:44.2953654Z skip: Need at least 2 CUDA devices (1.714s)
2023-01-11T21:26:44.2953907Z   test_fsdp_device_id_use_index_True (__main__.TestFSDPMisc)
2023-01-11T21:26:44.2954252Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2035
2023-01-11T21:26:44.2954605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2036
2023-01-11T21:26:44.2955059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2955385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2955808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2956281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2956617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.2957071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2957401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2957828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2958172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2958520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.2959029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2959545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2959912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.2960252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.2960555Z dist init r=1, world=2
2023-01-11T21:26:44.2960737Z dist init r=0, world=2
2023-01-11T21:26:44.2960934Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:26:44.2961434Z   test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2063
2023-01-11T21:26:44.2961868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2064
2023-01-11T21:26:44.2962314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2962648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2963079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2963429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2963757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.2964233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2964564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2964992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2965326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2965664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.2966163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2966668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2967061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.2967415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.2967678Z dist init r=1, world=2
2023-01-11T21:26:44.2967848Z dist init r=0, world=2
2023-01-11T21:26:44.2968058Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:26:44.2968480Z   test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2091
2023-01-11T21:26:44.2968918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2092
2023-01-11T21:26:44.2969372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2969705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2970136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2970476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2970816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.2971285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2971608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2972075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2972420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2972751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.2973232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2973833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2974218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.2974565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.2974812Z dist init r=0, world=2
2023-01-11T21:26:44.2974992Z dist init r=1, world=2
2023-01-11T21:26:44.2975199Z skip: Need at least 2 CUDA devices (1.714s)
2023-01-11T21:26:44.2975584Z   test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2119
2023-01-11T21:26:44.2976015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2120
2023-01-11T21:26:44.2976469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2976805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2977219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2977563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2977894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.2978350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2978675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2979103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2979447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2979770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.2980266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2980784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2981169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.2981499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.2981892Z dist init r=1, world=2
2023-01-11T21:26:44.2982089Z dist init r=0, world=2
2023-01-11T21:26:44.2982285Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:26:44.2982706Z   test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2147
2023-01-11T21:26:44.2983162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2148
2023-01-11T21:26:44.2983621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2983940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2984367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2984712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2985032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.2985500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2985916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2986348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2986720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2987058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.2987552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2988071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2988445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.2988788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.2989046Z dist init r=0, world=2
2023-01-11T21:26:44.2989212Z dist init r=1, world=2
2023-01-11T21:26:44.2989420Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:26:44.2989764Z   test_fsdp_namedtuple (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2175
2023-01-11T21:26:44.2990146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2176
2023-01-11T21:26:44.2990594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2990927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2991359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2991694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2992026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.2992499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2992825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2993237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2993580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2993913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.2994396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2994910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.2995295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.2995643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.2995889Z dist init r=1, world=2
2023-01-11T21:26:44.2996070Z dist init r=0, world=2
2023-01-11T21:26:44.2996276Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:26:44.2996611Z   test_fsdp_not_all_outputs_used_in_loss (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2203
2023-01-11T21:26:44.2996996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2204
2023-01-11T21:26:44.2997453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2997778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.2998256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.2998653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.2998983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.2999509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.2999828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.3000253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.3000594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.3000911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.3001482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.3002021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.3002416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.3002751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.3003009Z dist init r=1, world=2
2023-01-11T21:26:44.3003197Z dist init r=0, world=2
2023-01-11T21:26:44.3003392Z skip: Need at least 2 CUDA devices (1.715s)
2023-01-11T21:26:44.3003648Z   test_fsdp_same_model_across_ranks (__main__.TestFSDPMisc)
2023-01-11T21:26:44.3004028Z FSDP broadcasts model from rank 0 to ensure it starts off with the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2231
2023-01-11T21:26:44.3004428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2232
2023-01-11T21:26:44.3004865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.3005195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.3005625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.3005956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.3006287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.3006755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.3007078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.3007494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.3007836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.3008170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.3008669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.3009174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.3009561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.3009908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.3010154Z dist init r=1, world=2
2023-01-11T21:26:44.3010336Z dist init r=0, world=2
2023-01-11T21:26:44.3010540Z skip: Need at least 2 CUDA devices (1.714s)
2023-01-11T21:26:44.3010785Z   test_homogeneous_attributes (__main__.TestFSDPMisc)
2023-01-11T21:26:44.3011153Z Tests that passing heterogeneous values for attributes designated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2259
2023-01-11T21:26:44.3011613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2260
2023-01-11T21:26:44.3012102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.3012421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.3012852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.3013203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.3013541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.3013998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.3014323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.3014751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.3015083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.3015415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.3015918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.3016438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.3016815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.3017163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.3017422Z dist init r=0, world=2
2023-01-11T21:26:44.3017612Z dist init r=1, world=2
2023-01-11T21:26:44.3017815Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T21:26:44.3018079Z   test_module_device_mismatches_device_id (__main__.TestFSDPMisc)
2023-01-11T21:26:44.3018464Z Tests that specifying a ``device_id`` argument to FSDP for a GPU ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2287
2023-01-11T21:26:44.3018848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2288
2023-01-11T21:26:44.3019304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.3019641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.3020072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.3020403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.3020738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.3021212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.3021534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.3022120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.3022469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.3022805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.3023291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.3023810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.3024200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.3024663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.3024913Z dist init r=1, world=2
2023-01-11T21:26:44.3025140Z dist init r=0, world=2
2023-01-11T21:26:44.3025357Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:26:44.3025706Z   test_multi_device_not_supported (__main__.TestFSDPMisc)
2023-01-11T21:26:44.3026192Z Tests that wrapping a multi-device module (i.e. with submodules on ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2315
2023-01-11T21:26:44.3026594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2316
2023-01-11T21:26:44.3027036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.3027366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.3027798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.3028148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.3028476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.3028947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.3029281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.3029705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.3030036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.3030368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.3030866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.3031378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.3031765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.3032117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.3032374Z dist init r=1, world=2
2023-01-11T21:26:44.3032542Z dist init r=0, world=2
2023-01-11T21:26:44.3032751Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:26:44.3032978Z   test_no_params (__main__.TestFSDPMisc)
2023-01-11T21:26:44.3033313Z Test that device_id and cpu init work if module has no params ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2343
2023-01-11T21:26:44.3033701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2344
2023-01-11T21:26:44.3034153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.3034483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.3034900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.3035243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.3035579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.3036036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.3036359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.3036784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.3037125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.3037501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:26:44.3038027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.3038547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:26:44.3038930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:26:44.3039264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.3039523Z dist init r=1, world=2
2023-01-11T21:26:44.3039707Z dist init r=0, world=2
2023-01-11T21:26:44.3039903Z skip: Need at least 2 CUDA devices (1.713s)
2023-01-11T21:26:44.3040189Z   test_world_size_1_sharding_strategy_warning (__main__.TestFSDPMiscWorldSize1)
2023-01-11T21:26:44.3040597Z Tests that FSDP issues a warning when it switches to using ``NO_SHARD`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2371
2023-01-11T21:26:44.3041201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:26:44.3041533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:26:44.3041967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:26:44.3042313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:26:44.3042641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:26:44.3043137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:26:44.3043530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:26:44.3043792Z dist init r=0, world=1
2023-01-11T21:26:44.3043984Z skip: Need at least 1 CUDA device (1.611s)
2023-01-11T21:26:44.3044122Z 
2023-01-11T21:26:44.3044327Z ----------------------------------------------------------------------
2023-01-11T21:26:44.3044568Z Ran 18 tests in 30.700s
2023-01-11T21:26:44.3044683Z 
2023-01-11T21:26:44.3044743Z OK (skipped=18)
2023-01-11T21:26:44.3044852Z 
2023-01-11T21:26:44.3044937Z Generating XML reports...
2023-01-11T21:26:44.3045369Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20230111212613.xml
2023-01-11T21:26:44.3045950Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMiscWorldSize1-20230111212613.xml
2023-01-11T21:26:44.3046206Z 
2023-01-11T21:26:44.3046660Z ##[endgroup]
2023-01-11T21:26:44.3047103Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_misc_yo36j1on)
2023-01-11T21:26:44.3047364Z 
2023-01-11T21:26:44.3047593Z Running distributed/optim/test_zero_redundancy_optimizer ... [2023-01-11 21:26:44.291438]
2023-01-11T21:26:44.3048147Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:26:44.291932]
2023-01-11T21:27:31.1294658Z 
2023-01-11T21:27:31.1296041Z Expand the folded group to see the log file of distributed/optim/test_zero_redundancy_optimizer
2023-01-11T21:27:31.1297301Z ##[group]PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_d8ythz_a)
2023-01-11T21:27:31.1298420Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer
2023-01-11T21:27:31.1298821Z 
2023-01-11T21:27:31.1298951Z Running tests...
2023-01-11T21:27:31.1299493Z ----------------------------------------------------------------------
2023-01-11T21:27:31.1305457Z   test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1306595Z Check that ZeroRedundancyOptimizer properly handles adding a new ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2396
2023-01-11T21:27:31.1307283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2397
2023-01-11T21:27:31.1309014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1309450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1310589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1311238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1311813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1312277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1312613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1313043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1313389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1313698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1314065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:31.1314433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:31.1314932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:31.1315446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:31.1315740Z ok (2.077s)
2023-01-11T21:27:31.1316006Z   test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1316418Z Check the state consolidation mechanism and the state dict exposed ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2424
2023-01-11T21:27:31.1316815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2425
2023-01-11T21:27:31.1317272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1338053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1338842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1339507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1340112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1340981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1341315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1341922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1342277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1342734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1343253Z skip: CUDA is not available. (1.815s)
2023-01-11T21:27:31.1344011Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1344932Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1345902Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1346808Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1347729Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1348388Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1348885Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1349637Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1350488Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1351400Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1352290Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1353157Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1354089Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1354741Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1409300Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1410108Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1410823Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1411615Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1412408Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1413202Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1414048Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1414870Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1415712Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1416698Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1417594Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1418425Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1419451Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1420284Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1420987Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1421991Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1422857Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1423739Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1424451Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1425187Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2446
2023-01-11T21:27:31.1425727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2447
2023-01-11T21:27:31.1426440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1426947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1427638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1428175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1428644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1429398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1429900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1430619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1431187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1431712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1432105Z skip: CUDA is not available. (1.816s)
2023-01-11T21:27:31.1432605Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1433354Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2468
2023-01-11T21:27:31.1433969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2469
2023-01-11T21:27:31.1434736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1435298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1436231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1437000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1437580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1438341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1438904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1439660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1440260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1440829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1441416Z skip: CUDA is not available. (1.816s)
2023-01-11T21:27:31.1442029Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1442836Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2490
2023-01-11T21:27:31.1443482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2491
2023-01-11T21:27:31.1444275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1444823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1445570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1446131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1446670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1447508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1448096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1448857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1449457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1449958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1450404Z skip: CUDA is not available. (1.816s)
2023-01-11T21:27:31.1450959Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1451746Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2512
2023-01-11T21:27:31.1452421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2513
2023-01-11T21:27:31.1453238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1453767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1454520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1455122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1455623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1456456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1457015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1457722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1458424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1458805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1459090Z skip: CUDA is not available. (1.816s)
2023-01-11T21:27:31.1459422Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1459887Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2534
2023-01-11T21:27:31.1460279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2535
2023-01-11T21:27:31.1460747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1461065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1461497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1462179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1462492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1462956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1463283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1463710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1464039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1464361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1464642Z skip: CUDA is not available. (1.817s)
2023-01-11T21:27:31.1464981Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1465429Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2556
2023-01-11T21:27:31.1465822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2557
2023-01-11T21:27:31.1466271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1466585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1467009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1467352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1467678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1468126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1468451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1468873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1469214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1469517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1469797Z skip: CUDA is not available. (1.815s)
2023-01-11T21:27:31.1470082Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1470468Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2578
2023-01-11T21:27:31.1470932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2579
2023-01-11T21:27:31.1471425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1471755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1472172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1472518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1472840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1473287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1473614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1474038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1474382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1474690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1474967Z skip: CUDA is not available. (1.814s)
2023-01-11T21:27:31.1475262Z   test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1475682Z Check parity between constructing ZeRO with multiple parameter groups ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2600
2023-01-11T21:27:31.1476081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2601
2023-01-11T21:27:31.1476534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1476858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1477269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1477612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1477936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1478392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1478704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1479129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1479469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1479782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1480058Z skip: CUDA is not available. (1.814s)
2023-01-11T21:27:31.1480364Z   test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1480996Z Check that ZeroRedundancyOptimizer works with a non-default process ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2622
2023-01-11T21:27:31.1481401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2623
2023-01-11T21:27:31.1481852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1482181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1482595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1482938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1483260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1483653Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2023-01-11T21:27:31.1484209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1484535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1484960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1485301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1485607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1485994Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2023-01-11T21:27:31.1486274Z ok (1.815s)
2023-01-11T21:27:31.1486514Z   test_sharding (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1487048Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2644
2023-01-11T21:27:31.1487458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2645
2023-01-11T21:27:31.1487905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1488216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1488640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1488982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1489290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1489747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1490075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1490501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1490829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1491148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1491506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:31.1491869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:31.1492346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:31.1492862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:31.1493149Z ok (1.814s)
2023-01-11T21:27:31.1493387Z   test_step (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1493803Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2672
2023-01-11T21:27:31.1494208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2673
2023-01-11T21:27:31.1494656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1494970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1495395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1495737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1496044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1496541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1496896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1497322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1497650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1497970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1498252Z skip: CUDA is not available. (1.814s)
2023-01-11T21:27:31.1498532Z   test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1498957Z Check that ZeroRedundancyOptimizer properly exposes the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2694
2023-01-11T21:27:31.1499357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2695
2023-01-11T21:27:31.1499810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1500128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1500554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1500896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1501218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1501799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1502143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1502575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1502911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1503233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1503511Z skip: CUDA is not available. (1.814s)
2023-01-11T21:27:31.1503801Z   test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1504199Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2716
2023-01-11T21:27:31.1504591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2717
2023-01-11T21:27:31.1505043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1505357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1505781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1506127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1506448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1506890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1507216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1507643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1507983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1508289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1508646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:31.1509084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:31.1509614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:31.1510138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:31.1510528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:27:31.1510883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:27:31.1511357Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp669fbrv9'>
2023-01-11T21:27:31.1511692Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T21:27:31.1512121Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp9_x1kbot'>
2023-01-11T21:27:31.1512453Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T21:27:31.1512645Z ok (1.913s)
2023-01-11T21:27:31.1512902Z   test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1513284Z Check that the ZeRO join hook allows training with uneven inputs ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:27:31.1513688Z   test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1514238Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2744
2023-01-11T21:27:31.1514627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2745
2023-01-11T21:27:31.1515077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1515397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1515826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1516172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1516480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1516931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1517254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1517681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1518013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1518333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1518622Z skip: Need at least 4 CUDA devices (1.813s)
2023-01-11T21:27:31.1518950Z   test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed)
2023-01-11T21:27:31.1519498Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2766
2023-01-11T21:27:31.1519885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2767
2023-01-11T21:27:31.1520337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1520651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1521140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1521484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1521803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:31.1522289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1522647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1523075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1523405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1523727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1524015Z skip: Need at least 4 CUDA devices (1.813s)
2023-01-11T21:27:31.1524308Z   test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank)
2023-01-11T21:27:31.1524722Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2788
2023-01-11T21:27:31.1525254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1525583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1525998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1526338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1526658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1527019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:31.1527502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:27:31.1527788Z ok (1.710s)
2023-01-11T21:27:31.1528043Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank)
2023-01-11T21:27:31.1528433Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2802
2023-01-11T21:27:31.1528937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1529262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1529685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1530010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1530328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1530687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:31.1531179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:27:31.1531456Z ok (1.709s)
2023-01-11T21:27:31.1531721Z   test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank)
2023-01-11T21:27:31.1532158Z Check that ZeroRedundancyOptimizer raises an exception if the input ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2816
2023-01-11T21:27:31.1532673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1533001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1533427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1533770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1534077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1534435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:31.1534971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:27:31.1535308Z ok (1.709s)
2023-01-11T21:27:31.1535550Z   test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank)
2023-01-11T21:27:31.1535977Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2830
2023-01-11T21:27:31.1536508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1536825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1537250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1537590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1537913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1538261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:31.1538753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:27:31.1539043Z ok (1.711s)
2023-01-11T21:27:31.1539299Z   test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank)
2023-01-11T21:27:31.1539733Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2844
2023-01-11T21:27:31.1540257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1540581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1540997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1541341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1541785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1542161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:31.1542649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:27:31.1542937Z ok (1.810s)
2023-01-11T21:27:31.1543203Z   test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank)
2023-01-11T21:27:31.1543606Z Check that the ``step(**kwargs)`` interface is properly exposed. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2858
2023-01-11T21:27:31.1544118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1544445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1544874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1545206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1545532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1545893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:31.1546373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:27:31.1546663Z ok (1.709s)
2023-01-11T21:27:31.1546926Z   test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank)
2023-01-11T21:27:31.1547335Z Check that the ``step()`` method (without closure) is handled as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2872
2023-01-11T21:27:31.1547899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1548273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1548704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1549047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1549356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1549720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:31.1550220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:27:31.1550496Z ok (1.709s)
2023-01-11T21:27:31.1550753Z   test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank)
2023-01-11T21:27:31.1551155Z Check that the ``zero_grad`` method is properly handled. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2886
2023-01-11T21:27:31.1551667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:27:31.1551981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:31.1552435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:31.1552774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:31.1553092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:31.1553437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:31.1553928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:27:31.1554220Z ok (1.709s)
2023-01-11T21:27:31.1554325Z 
2023-01-11T21:27:31.1554516Z ----------------------------------------------------------------------
2023-01-11T21:27:31.1554760Z Ran 42 tests in 45.009s
2023-01-11T21:27:31.1554876Z 
2023-01-11T21:27:31.1554946Z OK (skipped=30)
2023-01-11T21:27:31.1555052Z 
2023-01-11T21:27:31.1555136Z Generating XML reports...
2023-01-11T21:27:31.1555677Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20230111212645.xml
2023-01-11T21:27:31.1556438Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20230111212645.xml
2023-01-11T21:27:31.1556768Z 
2023-01-11T21:27:31.1557171Z ##[endgroup]
2023-01-11T21:27:31.1557665Z FINISHED PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_d8ythz_a)
2023-01-11T21:27:31.1557961Z 
2023-01-11T21:27:31.1558146Z Running distributed/test_c10d_pypg ... [2023-01-11 21:27:31.130300]
2023-01-11T21:27:31.1558645Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:27:31.130694]
2023-01-11T21:28:30.9457500Z 
2023-01-11T21:28:30.9459459Z Expand the folded group to see the log file of distributed/test_c10d_pypg
2023-01-11T21:28:30.9460174Z ##[group]PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_pypg_f2rl9rr3)
2023-01-11T21:28:30.9460760Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_pypg
2023-01-11T21:28:30.9460961Z 
2023-01-11T21:28:30.9461037Z Running tests...
2023-01-11T21:28:30.9465478Z ----------------------------------------------------------------------
2023-01-11T21:28:30.9466310Z   test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkSubclass)
2023-01-11T21:28:30.9467532Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2911
2023-01-11T21:28:30.9468447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9468785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9469218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9469568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9469880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9470175Z skip: Need at least 2 CUDA devices (1.778s)
2023-01-11T21:28:30.9470470Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkSubclass)
2023-01-11T21:28:30.9470885Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2922
2023-01-11T21:28:30.9471408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9471736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9472165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9472498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9472822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9473110Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9473392Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2023-01-11T21:28:30.9473808Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2933
2023-01-11T21:28:30.9474324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9474652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9475068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9475419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9475744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9476032Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9476310Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2023-01-11T21:28:30.9476718Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2944
2023-01-11T21:28:30.9477230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9477551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9477976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9478320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9478645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9478918Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9479235Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2023-01-11T21:28:30.9479768Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2955
2023-01-11T21:28:30.9480367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9480704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9481219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9481565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9481877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9482169Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9482480Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2023-01-11T21:28:30.9483012Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2966
2023-01-11T21:28:30.9483524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9483855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9484290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9484622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9484941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9485230Z skip: Need at least 2 CUDA devices (1.508s)
2023-01-11T21:28:30.9485524Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2023-01-11T21:28:30.9486048Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2977
2023-01-11T21:28:30.9486578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9486908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9487339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9487672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9487997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9488290Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9488581Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2023-01-11T21:28:30.9489120Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2988
2023-01-11T21:28:30.9489649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9489982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9490400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9490749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9491075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9491357Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9491655Z   test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkSubclass)
2023-01-11T21:28:30.9492076Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2999
2023-01-11T21:28:30.9492597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9492961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9493426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9493816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9494180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9494591Z skip: Need at least 2 CUDA devices (1.508s)
2023-01-11T21:28:30.9495044Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2023-01-11T21:28:30.9495656Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3010
2023-01-11T21:28:30.9496394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9496950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9497694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9498293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9498825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9499283Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9499804Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2023-01-11T21:28:30.9500544Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3021
2023-01-11T21:28:30.9501368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9502081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9502710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9503167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9503602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9503997Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9504395Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2023-01-11T21:28:30.9504970Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3032
2023-01-11T21:28:30.9505663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9506110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9506707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9507255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9507755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9508215Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9508675Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2023-01-11T21:28:30.9509232Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3043
2023-01-11T21:28:30.9509935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9510370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9511139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9511608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9512128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9512512Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9513013Z   test_ddp_invoke_work_object (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3054
2023-01-11T21:28:30.9513745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9514197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9514832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9515364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9515862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9516442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ru_ihxb
2023-01-11T21:28:30.9517092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ru_ihxb/_remote_module_non_scriptable.py
2023-01-11T21:28:30.9517555Z ok (1.509s)
2023-01-11T21:28:30.9518022Z   test_ddp_with_pypg (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3065
2023-01-11T21:28:30.9518804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9519336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9519934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9520385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9520826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9521453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmh1hctqy
2023-01-11T21:28:30.9521996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmh1hctqy/_remote_module_non_scriptable.py
2023-01-11T21:28:30.9522507Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:28:30.9522852Z ok (1.608s)
2023-01-11T21:28:30.9523333Z   test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3076
2023-01-11T21:28:30.9524185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9524679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9525363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9525951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9526414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9526941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6o5_wqs
2023-01-11T21:28:30.9527507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6o5_wqs/_remote_module_non_scriptable.py
2023-01-11T21:28:30.9528050Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:28:30.9528391Z ok (1.608s)
2023-01-11T21:28:30.9528852Z   test_invalid_powerSGD_state (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3087
2023-01-11T21:28:30.9529636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9530230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9530961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9531520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9532025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9532980Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:30.9534526Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:30.9536017Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:30.9537392Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:30.9538811Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:30.9540200Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:30.9541065Z ok (1.509s)
2023-01-11T21:28:30.9541781Z   test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3098
2023-01-11T21:28:30.9542719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9543320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9544049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9544654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9545215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9545723Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9546384Z   test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3109
2023-01-11T21:28:30.9547337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9547913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9548871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9549591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9550166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9550669Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9551180Z   test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkWrapper)
2023-01-11T21:28:30.9552109Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3120
2023-01-11T21:28:30.9553009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9553590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9554334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9554943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9555549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9556071Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9556583Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkWrapper)
2023-01-11T21:28:30.9557321Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3131
2023-01-11T21:28:30.9558281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9558877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9559603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9560233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9560815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9561493Z skip: Need at least 2 CUDA devices (1.609s)
2023-01-11T21:28:30.9561994Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2023-01-11T21:28:30.9562737Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3142
2023-01-11T21:28:30.9563672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9564241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9565012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9565651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9566230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9566749Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9567278Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2023-01-11T21:28:30.9568006Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3153
2023-01-11T21:28:30.9568937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9569525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9570323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9570954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9571636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9572145Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9572801Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2023-01-11T21:28:30.9573807Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3164
2023-01-11T21:28:30.9574731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9575329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9576097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9576689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9577284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9577798Z skip: Need at least 2 CUDA devices (1.609s)
2023-01-11T21:28:30.9578321Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2023-01-11T21:28:30.9579263Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3175
2023-01-11T21:28:30.9580138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9580684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9581462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9582241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9582822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9583326Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9583814Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2023-01-11T21:28:30.9584787Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3186
2023-01-11T21:28:30.9585725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9586329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9587089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9587683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9588259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9588766Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9589296Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2023-01-11T21:28:30.9590289Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3197
2023-01-11T21:28:30.9591212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9591775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9592547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9593156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9593741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9594248Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9594915Z   test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkWrapper)
2023-01-11T21:28:30.9595752Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3208
2023-01-11T21:28:30.9596733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9597324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9598131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9598789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9599341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9599866Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9600516Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2023-01-11T21:28:30.9601311Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3219
2023-01-11T21:28:30.9602326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9602884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9603659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9604275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9604832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9605349Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9605890Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2023-01-11T21:28:30.9606648Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3230
2023-01-11T21:28:30.9607629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9608223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9608983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9609579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9610152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9610654Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9611196Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2023-01-11T21:28:30.9611925Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3241
2023-01-11T21:28:30.9612839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9613418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9614206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9614812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9615386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9615902Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9616424Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2023-01-11T21:28:30.9617272Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3252
2023-01-11T21:28:30.9618272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9618871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9619639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9620268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9620840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9621323Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9622117Z   test_ddp_invoke_work_object (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3263
2023-01-11T21:28:30.9623070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9623620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9624372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9624988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9625558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9626224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6vxuq4b
2023-01-11T21:28:30.9626939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6vxuq4b/_remote_module_non_scriptable.py
2023-01-11T21:28:30.9627435Z ok (1.609s)
2023-01-11T21:28:30.9627997Z   test_ddp_with_pypg (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3274
2023-01-11T21:28:30.9628906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9629513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9630331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9630962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9631525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9632190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgi168gl3
2023-01-11T21:28:30.9632912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgi168gl3/_remote_module_non_scriptable.py
2023-01-11T21:28:30.9633603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:28:30.9634081Z ok (1.509s)
2023-01-11T21:28:30.9634685Z   test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3285
2023-01-11T21:28:30.9635652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9636211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9636981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9637583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9638133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9638800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphvp6nz9v
2023-01-11T21:28:30.9639532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphvp6nz9v/_remote_module_non_scriptable.py
2023-01-11T21:28:30.9640358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:28:30.9640818Z ok (1.609s)
2023-01-11T21:28:30.9641598Z   test_invalid_powerSGD_state (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3296
2023-01-11T21:28:30.9642572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9643142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9643907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9644529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9645090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9646126Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:30.9647604Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:30.9648964Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:30.9650430Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:30.9651891Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:30.9653269Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:30.9654097Z ok (1.509s)
2023-01-11T21:28:30.9654690Z   test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3307
2023-01-11T21:28:30.9655664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9656227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9656966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9657569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9658121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9658632Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9659414Z   test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3318
2023-01-11T21:28:30.9660419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:30.9661003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:30.9661944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:30.9662556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:30.9663104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:30.9663599Z skip: Need at least 2 CUDA devices (1.509s)
2023-01-11T21:28:30.9663838Z 
2023-01-11T21:28:30.9664209Z ----------------------------------------------------------------------
2023-01-11T21:28:30.9664615Z Ran 38 tests in 58.209s
2023-01-11T21:28:30.9664819Z 
2023-01-11T21:28:30.9664946Z OK (skipped=30)
2023-01-11T21:28:30.9665130Z 
2023-01-11T21:28:30.9665271Z Generating XML reports...
2023-01-11T21:28:30.9666093Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20230111212732.xml
2023-01-11T21:28:30.9667174Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20230111212732.xml
2023-01-11T21:28:30.9667650Z 
2023-01-11T21:28:30.9668194Z ##[endgroup]
2023-01-11T21:28:30.9668975Z FINISHED PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_pypg_f2rl9rr3)
2023-01-11T21:28:30.9669414Z 
2023-01-11T21:28:30.9669773Z Running distributed/rpc/test_tensorpipe_agent ... [2023-01-11 21:28:30.946160]
2023-01-11T21:28:30.9670789Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:28:30.946615]
2023-01-11T22:03:09.9284775Z 
2023-01-11T22:03:09.9285351Z Expand the folded group to see the log file of distributed/rpc/test_tensorpipe_agent
2023-01-11T22:03:09.9287023Z ##[group]PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_g_gm984m)
2023-01-11T22:03:09.9320022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb0rn8y4i
2023-01-11T22:03:09.9320712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb0rn8y4i/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9322445Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_comparison>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_comparison_uneven_inputs>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_dist_autograd_sparse_grads>]>
2023-01-11T22:03:09.9323530Z test_ddp_comparison (__main__.TensorPipeDdpComparisonTest)
2023-01-11T22:03:09.9324046Z test_ddp_comparison_uneven_inputs (__main__.TensorPipeDdpComparisonTest)
2023-01-11T22:03:09.9324561Z test_ddp_dist_autograd_local_vs_remote (__main__.TensorPipeDdpComparisonTest)
2023-01-11T22:03:09.9325087Z test_ddp_dist_autograd_sparse_grads (__main__.TensorPipeDdpComparisonTest)
2023-01-11T22:03:09.9408432Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_inside>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_outside>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_outside_uneven_inputs>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_no_ddp>]>
2023-01-11T22:03:09.9409666Z test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTest)
2023-01-11T22:03:09.9410297Z test_backward_ddp_outside (__main__.TensorPipeDdpUnderDistAutogradTest)
2023-01-11T22:03:09.9411297Z test_backward_ddp_outside_uneven_inputs (__main__.TensorPipeDdpUnderDistAutogradTest)
2023-01-11T22:03:09.9411959Z test_backward_no_ddp (__main__.TensorPipeDdpUnderDistAutogradTest)
2023-01-11T22:03:09.9420402Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeDistAutogradTest testMethod=test_async_dist_autograd>, <__main__.TensorPipeDistAutogradTest testMethod=test_autograd_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_accumulate_grads>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_autograd_engine_error>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_complex_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_different_dtypes>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_different_tensor_dims>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_invalid_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_output_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_roots>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_round_trips>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_no_grad_on_tensor>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_node_failure>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_node_failure_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_python_udf_error>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref_multi>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref_nested>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_script_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_self>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_unused_send_function>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_unused_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_verify_hooks>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_without_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_without_rpc>, <__main__.TensorPipeDistAutogradTest testMethod=test_backwards_nested_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_clean_context_during_backward>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_nested_rpc>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_no_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_tensor_no_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_debug_info>, <__main__.TensorPipeDistAutogradTest testMethod=test_dist_autograd_profiling>, <__main__.TensorPipeDistAutogradTest testMethod=test_error_in_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_copy_sparse_indices_extra_ref>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_only_on_return_value>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_only_on_return_value_remote>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_builtin_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_builtin_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_call_itself>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_itself>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_python_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_python_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_mixed_requires_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_multiple_backward>, <__main__.TensorPipeDistAutogradTest testMethod=test_multiple_backward_with_errors>, <__main__.TensorPipeDistAutogradTest testMethod=test_nested_backward_accumulate_grads>, <__main__.TensorPipeDistAutogradTest testMethod=test_nested_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_grad_copy>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_grad_copy_sparse>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_remote>, <__main__.TensorPipeDistAutogradTest testMethod=test_post_hooks>, <__main__.TensorPipeDistAutogradTest testMethod=test_remote_complex_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_rpc_complex_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_thread_local_context_id>, <__main__.TensorPipeDistAutogradTest testMethod=test_trainer_ps>, <__main__.TensorPipeDistAutogradTest testMethod=test_trainer_ps_torchscript_functions>, <__main__.TensorPipeDistAutogradTest testMethod=test_worker_ids_recorded>]>
2023-01-11T22:03:09.9427548Z test_async_dist_autograd (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9427938Z test_autograd_context (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9428385Z test_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9428914Z test_backward_autograd_engine_error (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9429397Z test_backward_complex_python_udf (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9429880Z test_backward_different_dtypes (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9430319Z test_backward_different_tensor_dims (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9430742Z test_backward_invalid_args (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9431159Z test_backward_multiple_output_tensors (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9431608Z test_backward_multiple_roots (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9433959Z test_backward_multiple_round_trips (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9434513Z test_backward_no_grad_on_tensor (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9435072Z test_backward_node_failure (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9435646Z test_backward_node_failure_python_udf (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9436212Z test_backward_python_udf_error (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9436721Z test_backward_rref (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9437270Z test_backward_rref_multi (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9437763Z test_backward_rref_nested (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9438493Z test_backward_simple (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9439100Z test_backward_simple_python_udf (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9439500Z test_backward_simple_script_call (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9439928Z test_backward_simple_self (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9440360Z test_backward_unused_send_function (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9440811Z test_backward_unused_tensors (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9441243Z test_backward_verify_hooks (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9441669Z test_backward_without_context (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9442086Z test_backward_without_rpc (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9442505Z test_backwards_nested_python_udf (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9442962Z test_clean_context_during_backward (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9443419Z test_context_cleanup_nested_rpc (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9443995Z test_context_cleanup_no_tensors (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9444516Z test_context_cleanup_tensor_no_grad (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9444998Z test_context_cleanup_tensor_with_grad (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9445439Z test_debug_info (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9445861Z test_dist_autograd_profiling (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9446285Z test_error_in_context (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9446706Z test_grad_copy_sparse_indices_extra_ref (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9447161Z test_grad_only_on_return_value (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9447598Z test_grad_only_on_return_value_remote (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9448036Z test_graph_for_builtin_call (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9448484Z test_graph_for_builtin_remote_call (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9448949Z test_graph_for_py_nested_call (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9449415Z test_graph_for_py_nested_call_itself (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9449892Z test_graph_for_py_nested_remote_call (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9450351Z test_graph_for_py_nested_remote_call_itself (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9450822Z test_graph_for_python_call (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9451282Z test_graph_for_python_remote_call (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9451739Z test_mixed_requires_grad (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9452164Z test_multiple_backward (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9452622Z test_multiple_backward_with_errors (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9453110Z test_nested_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9453550Z test_nested_context (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9453972Z test_no_grad_copy (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9454395Z test_no_grad_copy_sparse (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9454840Z test_no_graph_with_tensors_not_require_grad (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9455345Z test_no_graph_with_tensors_not_require_grad_remote (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9455811Z test_post_hooks (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9456241Z test_remote_complex_args (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9456649Z test_rpc_complex_args (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9457092Z test_thread_local_context_id (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9457519Z test_trainer_ps (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9457961Z test_trainer_ps_torchscript_functions (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9458427Z test_worker_ids_recorded (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:09.9459361Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_exception>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_exception_on_constructor>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_none_grads>]>
2023-01-11T22:03:09.9460240Z test_dist_optim (__main__.TensorPipeDistOptimizerTest)
2023-01-11T22:03:09.9460660Z test_dist_optim_exception (__main__.TensorPipeDistOptimizerTest)
2023-01-11T22:03:09.9461137Z test_dist_optim_exception_on_constructor (__main__.TensorPipeDistOptimizerTest)
2023-01-11T22:03:09.9461620Z test_dist_optim_none_grads (__main__.TensorPipeDistOptimizerTest)
2023-01-11T22:03:09.9462851Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeJitDistAutogradTest testMethod=test_dist_backward>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_get_gradients>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_jit_fork_within_context>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_restore_context_after_swtich_to_jit_thread>]>
2023-01-11T22:03:09.9463821Z test_dist_backward (__main__.TensorPipeJitDistAutogradTest)
2023-01-11T22:03:09.9464246Z test_get_gradients (__main__.TensorPipeJitDistAutogradTest)
2023-01-11T22:03:09.9464700Z test_jit_fork_within_context (__main__.TensorPipeJitDistAutogradTest)
2023-01-11T22:03:09.9465200Z test_restore_context_after_swtich_to_jit_thread (__main__.TensorPipeJitDistAutogradTest)
2023-01-11T22:03:09.9472363Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeJitRpcTest testMethod=test_add_done_callback>, <__main__.TensorPipeJitRpcTest testMethod=test_all_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_args_and_kwargs_contain_different_types>, <__main__.TensorPipeJitRpcTest testMethod=test_args_kwargs_are_neither_passed>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_remote_multi>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_simple>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_decorator_order>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_return_type>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_return_type_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_async_script_throw>, <__main__.TensorPipeJitRpcTest testMethod=test_async_script_udf>, <__main__.TensorPipeJitRpcTest testMethod=test_call_fork_in_jit_with_profiling>, <__main__.TensorPipeJitRpcTest testMethod=test_call_python_function_remotely_from_script_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_call_rpc_with_profiling>, <__main__.TensorPipeJitRpcTest testMethod=test_call_script_function_that_not_exists_remotely_from_script>, <__main__.TensorPipeJitRpcTest testMethod=test_call_script_function_that_raises_remotely_from_script>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_chain>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_simple>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_with_exception>, <__main__.TensorPipeJitRpcTest testMethod=test_create_local_script_class_rref_in_py>, <__main__.TensorPipeJitRpcTest testMethod=test_create_local_script_module_rref_in_py>, <__main__.TensorPipeJitRpcTest testMethod=test_create_script_module_on_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_future_passed_between_python_and_jit>, <__main__.TensorPipeJitRpcTest testMethod=test_future_python_annotation>, <__main__.TensorPipeJitRpcTest testMethod=test_kwargs_not_passed>, <__main__.TensorPipeJitRpcTest testMethod=test_less_than_needed_args_are_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_load_script_module_with_pickled_rref>, <__main__.TensorPipeJitRpcTest testMethod=test_local_rref_local_value>, <__main__.TensorPipeJitRpcTest testMethod=test_more_than_needed_args_are_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_my_script_module_with_rrefs>, <__main__.TensorPipeJitRpcTest testMethod=test_no_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_record_function_jit_end_callbacks_with_fork>, <__main__.TensorPipeJitRpcTest testMethod=test_record_function_on_caller_rpc_async>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_module>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_throw>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_udf>, <__main__.TensorPipeJitRpcTest testMethod=test_return_local_script_class_rref_in_py_and_use_in_script>, <__main__.TensorPipeJitRpcTest testMethod=test_return_local_script_module_rref_in_py_and_use_in_script>, <__main__.TensorPipeJitRpcTest testMethod=test_rpc_async_jit_profiled>, <__main__.TensorPipeJitRpcTest testMethod=test_rpc_torchscript_record_function>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_as_arg_and_return>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_is_owner>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_jit_pickle_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_list_mutate>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_local_value>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_python_annotation>, <__main__.TensorPipeJitRpcTest testMethod=test_some_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_function>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_function_exception>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_functions_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_unexepected_kwarg_is_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_user_rrefs_confirmed>, <__main__.TensorPipeJitRpcTest testMethod=test_user_rrefs_confirmed_remote>]>
2023-01-11T22:03:09.9479342Z test_add_done_callback (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9479793Z test_all_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9480274Z test_args_and_kwargs_contain_different_types (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9480732Z test_args_kwargs_are_neither_passed (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9481148Z test_async_function_remote (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9481570Z test_async_function_remote_multi (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9481998Z test_async_function_simple (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9482416Z test_async_function_wrong_decorator_order (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9482865Z test_async_function_wrong_return_type (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9483330Z test_async_function_wrong_return_type_remote (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9483772Z test_async_script_throw (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9484162Z test_async_script_udf (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9484583Z test_call_fork_in_jit_with_profiling (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9485074Z test_call_python_function_remotely_from_script_not_supported (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9485530Z test_call_rpc_with_profiling (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9486007Z test_call_script_function_that_not_exists_remotely_from_script (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9486524Z test_call_script_function_that_raises_remotely_from_script (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9486972Z test_callback_chain (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9487346Z test_callback_simple (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9487753Z test_callback_with_exception (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9488208Z test_create_local_script_class_rref_in_py (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9488657Z test_create_local_script_module_rref_in_py (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9489112Z test_create_script_module_on_remote (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9489564Z test_future_passed_between_python_and_jit (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9490003Z test_future_python_annotation (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9490401Z test_kwargs_not_passed (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9490829Z test_less_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9491294Z test_load_script_module_with_pickled_rref (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9491720Z test_local_rref_local_value (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9492155Z test_more_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9492601Z test_my_script_module_with_rrefs (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9493046Z test_no_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9493550Z test_record_function_jit_end_callbacks_with_fork (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9494020Z test_record_function_on_caller_rpc_async (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9494484Z test_remote_script_module (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9494876Z test_remote_script_throw (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9495278Z test_remote_script_udf (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9495734Z test_return_local_script_class_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9496237Z test_return_local_script_module_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9496691Z test_rpc_async_jit_profiled (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9497146Z test_rpc_torchscript_record_function (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9497568Z test_rref_as_arg_and_return (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9497938Z test_rref_is_owner (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9498340Z test_rref_jit_pickle_not_supported (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9498736Z test_rref_list_mutate (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9499127Z test_rref_local_value (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9499530Z test_rref_python_annotation (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9499977Z test_some_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9500408Z test_torchscript_function (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9500825Z test_torchscript_function_exception (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9501289Z test_torchscript_functions_not_supported (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9501859Z test_unexepected_kwarg_is_specified (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9502300Z test_user_rrefs_confirmed (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9502714Z test_user_rrefs_confirmed_remote (__main__.TensorPipeJitRpcTest)
2023-01-11T22:03:09.9503303Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeParameterServerTest testMethod=test_batch_updating_parameter_server>]>
2023-01-11T22:03:09.9503905Z test_batch_updating_parameter_server (__main__.TensorPipeParameterServerTest)
2023-01-11T22:03:09.9504495Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeReinforcementLearningRpcTest testMethod=test_rl_rpc>]>
2023-01-11T22:03:09.9505073Z test_rl_rpc (__main__.TensorPipeReinforcementLearningRpcTest)
2023-01-11T22:03:09.9507200Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeRemoteModuleTest testMethod=test_bad_module>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_async>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_async_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_sync>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_sync_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_with_kwargs>, <__main__.TensorPipeRemoteModuleTest testMethod=test_get_module_rref>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_module_py_pickle_not_supported>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_module_py_pickle_not_supported_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_parameters>, <__main__.TensorPipeRemoteModuleTest testMethod=test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire>, <__main__.TensorPipeRemoteModuleTest testMethod=test_train_eval>, <__main__.TensorPipeRemoteModuleTest testMethod=test_unsupported_methods>]>
2023-01-11T22:03:09.9509325Z test_bad_module (__main__.TensorPipeRemoteModuleTest)
2023-01-11T22:03:09.9509751Z test_forward_async (__main__.TensorPipeRemoteModuleTest)
2023-01-11T22:03:09.9510159Z test_forward_async_script (__main__.TensorPipeRemoteModuleTest)
2023-01-11T22:03:09.9510601Z test_forward_sync (__main__.TensorPipeRemoteModuleTest)
2023-01-11T22:03:09.9511027Z test_forward_sync_script (__main__.TensorPipeRemoteModuleTest)
2023-01-11T22:03:09.9511563Z test_forward_with_kwargs (__main__.TensorPipeRemoteModuleTest)
2023-01-11T22:03:09.9511994Z test_get_module_rref (__main__.TensorPipeRemoteModuleTest)
2023-01-11T22:03:09.9512512Z test_remote_module_py_pickle_not_supported (__main__.TensorPipeRemoteModuleTest)
2023-01-11T22:03:09.9513030Z test_remote_module_py_pickle_not_supported_script (__main__.TensorPipeRemoteModuleTest)
2023-01-11T22:03:09.9513493Z test_remote_parameters (__main__.TensorPipeRemoteModuleTest)
2023-01-11T22:03:09.9514006Z test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire (__main__.TensorPipeRemoteModuleTest)
2023-01-11T22:03:09.9514524Z test_train_eval (__main__.TensorPipeRemoteModuleTest)
2023-01-11T22:03:09.9514948Z test_unsupported_methods (__main__.TensorPipeRemoteModuleTest)
2023-01-11T22:03:09.9593892Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeRpcTest testMethod=test_add>, <__main__.TensorPipeRpcTest testMethod=test_add_done_callback>, <__main__.TensorPipeRpcTest testMethod=test_add_with_id>, <__main__.TensorPipeRpcTest testMethod=test_all_gather>, <__main__.TensorPipeRpcTest testMethod=test_all_gather_timeout>, <__main__.TensorPipeRpcTest testMethod=test_async_add>, <__main__.TensorPipeRpcTest testMethod=test_async_class_method>, <__main__.TensorPipeRpcTest testMethod=test_async_class_method_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy_async>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_chained>, <__main__.TensorPipeRpcTest testMethod=test_async_function_chained_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_nested>, <__main__.TensorPipeRpcTest testMethod=test_async_function_nested_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_simple>, <__main__.TensorPipeRpcTest testMethod=test_async_function_with_future_ctor>, <__main__.TensorPipeRpcTest testMethod=test_async_function_with_future_ctor_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_cbs_jit_call>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_double_end_callbacks>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_legacy>, <__main__.TensorPipeRpcTest testMethod=test_async_static_method>, <__main__.TensorPipeRpcTest testMethod=test_async_static_method_remote>, <__main__.TensorPipeRpcTest testMethod=test_build_rpc_profiling_key>, <__main__.TensorPipeRpcTest testMethod=test_builtin_remote_ret>, <__main__.TensorPipeRpcTest testMethod=test_builtin_remote_self>, <__main__.TensorPipeRpcTest testMethod=test_call_method_on_rref>, <__main__.TensorPipeRpcTest testMethod=test_callback_chain>, <__main__.TensorPipeRpcTest testMethod=test_callback_in_rpc>, <__main__.TensorPipeRpcTest testMethod=test_callback_multi>, <__main__.TensorPipeRpcTest testMethod=test_callback_none>, <__main__.TensorPipeRpcTest testMethod=test_callback_simple>, <__main__.TensorPipeRpcTest testMethod=test_callback_with_error>, <__main__.TensorPipeRpcTest testMethod=test_callback_with_ret>, <__main__.TensorPipeRpcTest testMethod=test_callback_wrong_arg_num>, <__main__.TensorPipeRpcTest testMethod=test_callback_wrong_arg_type>, <__main__.TensorPipeRpcTest testMethod=test_cannot_infer_backend_from_options>, <__main__.TensorPipeRpcTest testMethod=test_custom_exception_throw_during_reconstruction>, <__main__.TensorPipeRpcTest testMethod=test_deadlock>, <__main__.TensorPipeRpcTest testMethod=test_debug_info>, <__main__.TensorPipeRpcTest testMethod=test_default_timeout_used>, <__main__.TensorPipeRpcTest testMethod=test_disable_gil_profiling>, <__main__.TensorPipeRpcTest testMethod=test_dist_init_decorator>, <__main__.TensorPipeRpcTest testMethod=test_duplicate_name>, <__main__.TensorPipeRpcTest testMethod=test_duplicate_name_2>, <__main__.TensorPipeRpcTest testMethod=test_expected_src>, <__main__.TensorPipeRpcTest testMethod=test_function_not_on_callee>, <__main__.TensorPipeRpcTest testMethod=test_future_done>, <__main__.TensorPipeRpcTest testMethod=test_future_done_exception>, <__main__.TensorPipeRpcTest testMethod=test_future_in_rpc>, <__main__.TensorPipeRpcTest testMethod=test_future_nested_callback>, <__main__.TensorPipeRpcTest testMethod=test_future_wait_twice>, <__main__.TensorPipeRpcTest testMethod=test_get_worker_infos>, <__main__.TensorPipeRpcTest testMethod=test_graceful_shutdown_with_uneven_workload>, <__main__.TensorPipeRpcTest testMethod=test_handle_send_exceptions>, <__main__.TensorPipeRpcTest testMethod=test_ignore_rref_leak>, <__main__.TensorPipeRpcTest testMethod=test_init_pg_then_rpc>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_then_pg>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_twice>, <__main__.TensorPipeRpcTest testMethod=test_int_callee>, <__main__.TensorPipeRpcTest testMethod=test_invalid_names>, <__main__.TensorPipeRpcTest testMethod=test_local_rref_no_fork>, <__main__.TensorPipeRpcTest testMethod=test_local_shutdown>, <__main__.TensorPipeRpcTest testMethod=test_local_shutdown_with_rpc>, <__main__.TensorPipeRpcTest testMethod=test_local_value_not_on_owner>, <__main__.TensorPipeRpcTest testMethod=test_mark_future_twice>, <__main__.TensorPipeRpcTest testMethod=test_multi_builtin_remote_ret>, <__main__.TensorPipeRpcTest testMethod=test_multi_layer_nested_async_rpc>, <__main__.TensorPipeRpcTest testMethod=test_multi_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_multi_rpc>, <__main__.TensorPipeRpcTest testMethod=test_my_parameter_server>, <__main__.TensorPipeRpcTest testMethod=test_nested_remote>, <__main__.TensorPipeRpcTest testMethod=test_nested_rpc>, <__main__.TensorPipeRpcTest testMethod=test_nested_rref>, <__main__.TensorPipeRpcTest testMethod=test_nested_rref_stress>, <__main__.TensorPipeRpcTest testMethod=test_non_cont_tensors>, <__main__.TensorPipeRpcTest testMethod=test_non_garbage_collected_user_rref_due_to_local_circular_dependency>, <__main__.TensorPipeRpcTest testMethod=test_nonzero>, <__main__.TensorPipeRpcTest testMethod=test_owner_equality>, <__main__.TensorPipeRpcTest testMethod=test_owner_rref_backward>, <__main__.TensorPipeRpcTest testMethod=test_pass_local_rrefs>, <__main__.TensorPipeRpcTest testMethod=test_pg_init_no_rpc_init>, <__main__.TensorPipeRpcTest testMethod=test_pickle_future>, <__main__.TensorPipeRpcTest testMethod=test_profiler_export_trace>, <__main__.TensorPipeRpcTest testMethod=test_profiler_remote_events_profiled>, <__main__.TensorPipeRpcTest testMethod=test_profiler_remote_events_profiled_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_key_names>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_memory>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_record_shapes>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_autograd_context>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_autograd_context_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_async_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_async_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_remote_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_remote_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_sync_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_sync_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_py_built_in>, <__main__.TensorPipeRpcTest testMethod=test_py_class_constructor>, <__main__.TensorPipeRpcTest testMethod=test_py_class_instance_method>, <__main__.TensorPipeRpcTest testMethod=test_py_class_method>, <__main__.TensorPipeRpcTest testMethod=test_py_class_static_method>, <__main__.TensorPipeRpcTest testMethod=test_py_function_exception>, <__main__.TensorPipeRpcTest testMethod=test_py_multi_async_call>, <__main__.TensorPipeRpcTest testMethod=test_py_nested_pickle>, <__main__.TensorPipeRpcTest testMethod=test_py_no_return_result>, <__main__.TensorPipeRpcTest testMethod=test_py_raise_in_user_func>, <__main__.TensorPipeRpcTest testMethod=test_py_raise_in_user_func_escaped_str>, <__main__.TensorPipeRpcTest testMethod=test_py_rpc_rref_args>, <__main__.TensorPipeRpcTest testMethod=test_py_rref_args>, <__main__.TensorPipeRpcTest testMethod=test_py_rref_args_user_share>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors_in_container>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors_multi_async_call>, <__main__.TensorPipeRpcTest testMethod=test_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_py_user_defined>, <__main__.TensorPipeRpcTest testMethod=test_register_rpc_backend_and_set_and_start_rpc_backend>, <__main__.TensorPipeRpcTest testMethod=test_reinit>, <__main__.TensorPipeRpcTest testMethod=test_remote_same_worker>, <__main__.TensorPipeRpcTest testMethod=test_remote_throw>, <__main__.TensorPipeRpcTest testMethod=test_remote_with_exception>, <__main__.TensorPipeRpcTest testMethod=test_return_future>, <__main__.TensorPipeRpcTest testMethod=test_return_future_async>, <__main__.TensorPipeRpcTest testMethod=test_return_future_remote>, <__main__.TensorPipeRpcTest testMethod=test_return_local_rrefs>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_all>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_multithreaded>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_partial_subset>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_subset>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_async_function>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_async_function_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_remote_record_function>, <__main__.TensorPipeRpcTest testMethod=test_rpc_return_rref>, <__main__.TensorPipeRpcTest testMethod=test_rpc_timeouts>, <__main__.TensorPipeRpcTest testMethod=test_rref_context_debug_info>, <__main__.TensorPipeRpcTest testMethod=test_rref_forward_chain>, <__main__.TensorPipeRpcTest testMethod=test_rref_get_future>, <__main__.TensorPipeRpcTest testMethod=test_rref_leak>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_class>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_class_self>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_non_exist>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_reuse>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_tensor>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_tensor_self>, <__main__.TensorPipeRpcTest testMethod=test_rref_py_pickle_not_supported>, <__main__.TensorPipeRpcTest testMethod=test_rref_str>, <__main__.TensorPipeRpcTest testMethod=test_rref_timeout>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_owner_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_owner_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_slow_init>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_with_error_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_with_error_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_scalar_add>, <__main__.TensorPipeRpcTest testMethod=test_self_add>, <__main__.TensorPipeRpcTest testMethod=test_self_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_remote_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_rpc_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_self_remote_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_self_rpc_arg>, <__main__.TensorPipeRpcTest testMethod=test_send_to_rank>, <__main__.TensorPipeRpcTest testMethod=test_server_process_global_profiler>, <__main__.TensorPipeRpcTest testMethod=test_set_and_get_default_rpc_timeout>, <__main__.TensorPipeRpcTest testMethod=test_shutdown_errors>, <__main__.TensorPipeRpcTest testMethod=test_shutdown_followed_by_rpc>, <__main__.TensorPipeRpcTest testMethod=test_stress_heavy_rpc>, <__main__.TensorPipeRpcTest testMethod=test_stress_heavy_rpc_torchscript>, <__main__.TensorPipeRpcTest testMethod=test_stress_light_rpc>, <__main__.TensorPipeRpcTest testMethod=test_use_rpc_pickler>, <__main__.TensorPipeRpcTest testMethod=test_use_rref_after_shutdown>, <__main__.TensorPipeRpcTest testMethod=test_user_rref_backward>, <__main__.TensorPipeRpcTest testMethod=test_user_rrefs_confirmed>, <__main__.TensorPipeRpcTest testMethod=test_user_rrefs_confirmed_remote>, <__main__.TensorPipeRpcTest testMethod=test_wait_all>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_builtin>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_python>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_script_function>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_multiple_call>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_raise_in_body>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_raise_in_user_func>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_timeout>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_with_exception>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_with_partial_exception>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_dense>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_timeout>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_twice_dense>, <__main__.TensorPipeRpcTest testMethod=test_worker_id>, <__main__.TensorPipeRpcTest testMethod=test_worker_info_pickle>, <__main__.TensorPipeRpcTest testMethod=test_world_size_one>, <__main__.TensorPipeRpcTest testMethod=test_wrong_types>]>
2023-01-11T22:03:09.9627910Z test_add (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9649534Z test_add_done_callback (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9650259Z test_add_with_id (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9671769Z test_all_gather (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9672248Z test_all_gather_timeout (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9672711Z test_async_add (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9673173Z test_async_class_method (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9673940Z test_async_class_method_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9674409Z test_async_class_rref_proxy (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9674996Z test_async_class_rref_proxy_async (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9675520Z test_async_class_rref_proxy_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9675997Z test_async_function_chained (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9676506Z test_async_function_chained_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9677039Z test_async_function_multi_chained (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9677599Z test_async_function_multi_chained_async (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9678131Z test_async_function_multi_chained_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9678651Z test_async_function_multi_fanout (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9679213Z test_async_function_multi_fanout_async (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9679693Z test_async_function_multi_fanout_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9680196Z test_async_function_nested (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9680701Z test_async_function_nested_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9681253Z test_async_function_raise (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9681780Z test_async_function_raise_async (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9682350Z test_async_function_raise_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9682909Z test_async_function_simple (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9683463Z test_async_function_with_future_ctor (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9684059Z test_async_function_with_future_ctor_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9684673Z test_async_function_wrong_return_type (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9685257Z test_async_function_wrong_return_type_async (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9685903Z test_async_function_wrong_return_type_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9686517Z test_async_record_function_cbs_jit_call (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9687149Z test_async_record_function_double_end_callbacks (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9687731Z test_async_record_function_legacy (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9688267Z test_async_static_method (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9688798Z test_async_static_method_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9689314Z test_build_rpc_profiling_key (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9689851Z test_builtin_remote_ret (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9690358Z test_builtin_remote_self (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9690871Z test_call_method_on_rref (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9691356Z test_callback_chain (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9691856Z test_callback_in_rpc (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9692364Z test_callback_multi (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9692837Z test_callback_none (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9693331Z test_callback_simple (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9693841Z test_callback_with_error (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9694336Z test_callback_with_ret (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9694860Z test_callback_wrong_arg_num (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9695386Z test_callback_wrong_arg_type (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9695949Z test_cannot_infer_backend_from_options (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9696547Z test_custom_exception_throw_during_reconstruction (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9697096Z test_deadlock (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9697570Z test_debug_info (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9698059Z test_default_timeout_used (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9698756Z test_disable_gil_profiling (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9699279Z test_dist_init_decorator (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9699837Z test_duplicate_name (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9700340Z test_duplicate_name_2 (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9700850Z test_expected_src (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9701364Z test_function_not_on_callee (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9702117Z test_future_done (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9702629Z test_future_done_exception (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9703133Z test_future_in_rpc (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9703625Z test_future_nested_callback (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9704147Z test_future_wait_twice (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9704641Z test_get_worker_infos (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9705183Z test_graceful_shutdown_with_uneven_workload (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9705777Z test_handle_send_exceptions (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9706286Z test_ignore_rref_leak (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9706805Z test_init_pg_then_rpc (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9707286Z test_init_rpc_then_pg (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9707784Z test_init_rpc_twice (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9708270Z test_int_callee (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9708739Z test_invalid_names (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9709228Z test_local_rref_no_fork (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9709732Z test_local_shutdown (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9710224Z test_local_shutdown_with_rpc (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9710759Z test_local_value_not_on_owner (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9711285Z test_mark_future_twice (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9711811Z test_multi_builtin_remote_ret (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9712335Z test_multi_layer_nested_async_rpc (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9712880Z test_multi_py_udf_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9713372Z test_multi_rpc (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9713850Z test_my_parameter_server (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9714346Z test_nested_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9714837Z test_nested_rpc (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9715320Z test_nested_rref (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9715793Z test_nested_rref_stress (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9716298Z test_non_cont_tensors (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9716915Z test_non_garbage_collected_user_rref_due_to_local_circular_dependency (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9717481Z test_nonzero (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9717976Z test_owner_equality (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9718489Z test_owner_rref_backward (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9719044Z test_pass_local_rrefs (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9719568Z test_pg_init_no_rpc_init (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9720062Z test_pickle_future (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9720563Z test_profiler_export_trace (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9721095Z test_profiler_remote_events_profiled (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9721701Z test_profiler_remote_events_profiled_single_threaded (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9722282Z test_profiler_rpc_key_names (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9722782Z test_profiler_rpc_memory (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9723318Z test_profiler_rpc_record_shapes (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9723876Z test_profiler_with_async_rpc_builtin (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9724629Z test_profiler_with_async_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9725213Z test_profiler_with_async_rpc_udf (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9725873Z test_profiler_with_async_rpc_udf_single_threaded (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9726478Z test_profiler_with_autograd_context (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9727055Z test_profiler_with_autograd_context_single_threaded (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9727650Z test_profiler_with_remote_builtin (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9728252Z test_profiler_with_remote_builtin_single_threaded (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9728817Z test_profiler_with_remote_udf (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9729379Z test_profiler_with_remote_udf_single_threaded (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9729962Z test_profiler_with_script_async_rpc (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9730572Z test_profiler_with_script_async_rpc_single_threaded (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9731171Z test_profiler_with_script_remote_rpc (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9731783Z test_profiler_with_script_remote_rpc_single_threaded (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9732372Z test_profiler_with_script_sync_rpc (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9732949Z test_profiler_with_script_sync_rpc_single_threaded (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9733542Z test_profiler_with_sync_rpc_builtin (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9734146Z test_profiler_with_sync_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9734728Z test_profiler_with_sync_rpc_udf (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9735306Z test_profiler_with_sync_rpc_udf_single_threaded (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9735858Z test_py_built_in (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9736352Z test_py_class_constructor (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9736860Z test_py_class_instance_method (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9737363Z test_py_class_method (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9737876Z test_py_class_static_method (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9738386Z test_py_function_exception (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9738902Z test_py_multi_async_call (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9739414Z test_py_nested_pickle (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9739908Z test_py_no_return_result (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9740423Z test_py_raise_in_user_func (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9740975Z test_py_raise_in_user_func_escaped_str (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9741520Z test_py_rpc_rref_args (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9742142Z test_py_rref_args (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9742657Z test_py_rref_args_user_share (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9743155Z test_py_tensors (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9743646Z test_py_tensors_in_container (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9744176Z test_py_tensors_multi_async_call (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9744683Z test_py_udf_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9745154Z test_py_user_defined (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9745723Z test_register_rpc_backend_and_set_and_start_rpc_backend (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9746274Z test_reinit (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9746736Z test_remote_same_worker (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9747216Z test_remote_throw (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9747715Z test_remote_with_exception (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9748211Z test_return_future (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9748688Z test_return_future_async (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9749385Z test_return_future_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9749902Z test_return_local_rrefs (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9750379Z test_rpc_barrier_all (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9751009Z test_rpc_barrier_multithreaded (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9751570Z test_rpc_barrier_partial_subset (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9752079Z test_rpc_barrier_subset (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9752614Z test_rpc_profiling_async_function (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9753223Z test_rpc_profiling_async_function_single_threaded (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9753824Z test_rpc_profiling_remote_record_function (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9754356Z test_rpc_return_rref (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9754845Z test_rpc_timeouts (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9755342Z test_rref_context_debug_info (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9755831Z test_rref_forward_chain (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9756328Z test_rref_get_future (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9756812Z test_rref_leak (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9757274Z test_rref_proxy_class (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9757785Z test_rref_proxy_class_self (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9758294Z test_rref_proxy_non_exist (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9758770Z test_rref_proxy_reuse (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9759347Z test_rref_proxy_tensor (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9759856Z test_rref_proxy_tensor_self (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9760391Z test_rref_py_pickle_not_supported (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9760884Z test_rref_str (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9761357Z test_rref_timeout (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9761855Z test_rref_type_blocking (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9762363Z test_rref_type_non_blocking (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9762897Z test_rref_type_owner_blocking (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9763435Z test_rref_type_owner_non_blocking (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9763947Z test_rref_type_slow_init (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9764486Z test_rref_type_with_error_blocking (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9765044Z test_rref_type_with_error_non_blocking (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9765555Z test_scalar_add (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9766023Z test_self_add (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9766505Z test_self_py_udf_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9767031Z test_self_remote_rref_as_remote_arg (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9767563Z test_self_remote_rref_as_rpc_arg (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9768115Z test_self_remote_rref_as_self_remote_arg (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9768682Z test_self_remote_rref_as_self_rpc_arg (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9769171Z test_send_to_rank (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9769700Z test_server_process_global_profiler (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9770263Z test_set_and_get_default_rpc_timeout (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9770764Z test_shutdown_errors (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9771285Z test_shutdown_followed_by_rpc (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9771803Z test_stress_heavy_rpc (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9772335Z test_stress_heavy_rpc_torchscript (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9792544Z test_stress_light_rpc (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9793055Z test_use_rpc_pickler (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9793563Z test_use_rref_after_shutdown (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9794210Z test_user_rref_backward (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9794715Z test_user_rrefs_confirmed (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9795296Z test_user_rrefs_confirmed_remote (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9795777Z test_wait_all (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9796273Z test_wait_all_exit_early_builtin (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9796799Z test_wait_all_exit_early_python (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9797356Z test_wait_all_exit_early_script_function (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9797893Z test_wait_all_multiple_call (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9798409Z test_wait_all_raise_in_body (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9799018Z test_wait_all_raise_in_user_func (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9799507Z test_wait_all_timeout (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9800019Z test_wait_all_with_exception (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9800557Z test_wait_all_with_partial_exception (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9801078Z test_wait_all_workers_dense (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9801606Z test_wait_all_workers_timeout (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9802134Z test_wait_all_workers_twice_dense (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9802631Z test_worker_id (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9803091Z test_worker_info_pickle (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9803588Z test_world_size_one (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9804064Z test_wrong_types (__main__.TensorPipeRpcTest)
2023-01-11T22:03:09.9813039Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_different_dtypes_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_multiple_round_trips_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_no_grad_on_tensor_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_multi_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_nested_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_python_udf_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_script_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_self_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backwards_nested_python_udf_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_nested_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_tensor_no_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_embedding_bag_with_no_grad_tensors>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_builtin_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_builtin_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_call_itself_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_itself_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_python_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_python_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_mixed_requires_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_multiple_backward_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_nested_backward_accumulate_grads_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_remote_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_remote_complex_args_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_rpc_complex_args_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_trainer_ps_sparse>]>
2023-01-11T22:03:09.9822202Z test_backward_different_dtypes_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9822996Z test_backward_multiple_round_trips_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9823799Z test_backward_no_grad_on_tensor_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9824567Z test_backward_rref_multi_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9825334Z test_backward_rref_nested_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9826072Z test_backward_rref_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9826843Z test_backward_simple_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9827638Z test_backward_simple_script_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9828399Z test_backward_simple_self_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9829154Z test_backward_simple_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9829919Z test_backwards_nested_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9830706Z test_context_cleanup_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9831486Z test_context_cleanup_tensor_no_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9832290Z test_context_cleanup_tensor_with_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9833104Z test_embedding_bag_with_no_grad_tensors (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9833868Z test_graph_for_builtin_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9834651Z test_graph_for_builtin_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9835448Z test_graph_for_py_nested_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9836238Z test_graph_for_py_nested_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9837029Z test_graph_for_py_nested_remote_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9837849Z test_graph_for_py_nested_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9838629Z test_graph_for_python_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9839474Z test_graph_for_python_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9840234Z test_mixed_requires_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9840989Z test_multiple_backward_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9841775Z test_nested_backward_accumulate_grads_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9842594Z test_no_graph_with_tensors_not_require_grad_remote_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9843602Z test_no_graph_with_tensors_not_require_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9844465Z test_remote_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9845217Z test_rpc_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9845910Z test_trainer_ps_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2023-01-11T22:03:09.9855118Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_builtin_remote_ret_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_builtin_remote_self_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_and_static_init_rpc_together>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_existing_rank_can_communicate_with_new_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_init_rpc>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_init_rpc_without_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_new_rank_can_communicated_with_existing_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_infer_backend_from_options>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_mismatched_type_for_options>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_builtin_remote_ret_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_py_udf_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_my_parameter_server_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rref_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rref_stress_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_op_with_invalid_args>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rpc_rref_args_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rref_args_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rref_args_user_share_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_sparse_tensors_in_container>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_get_type_timeout_blocking>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_get_type_timeout_non_blocking>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_proxy_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_py_udf_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_remote_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_rpc_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_self_remote_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_self_rpc_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_send_to_rank_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_set_and_get_num_worker_threads>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_stress_heavy_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_tensorpipe_options_throw_on_timedelta_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_tensorpipe_set_default_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_wait_all_workers_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_wait_all_workers_twice_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_world_size_one_sparse>]>
2023-01-11T22:03:09.9864480Z test_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9865143Z test_builtin_remote_self_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9865952Z test_dynamic_and_static_init_rpc_together (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9866713Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9867500Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9868209Z test_dynamic_rpc_init_rpc (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9868878Z test_dynamic_rpc_init_rpc_without_rank (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9869617Z test_dynamic_rpc_new_rank_can_communicated_with_existing_rank (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9870324Z test_infer_backend_from_options (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9870987Z test_mismatched_type_for_options (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9871666Z test_multi_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9872313Z test_multi_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9872950Z test_multi_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9873588Z test_my_parameter_server_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9874231Z test_nested_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9874850Z test_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9875464Z test_nested_rref_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9876100Z test_nested_rref_stress_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9876724Z test_op_with_invalid_args (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9877372Z test_py_rpc_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9878007Z test_py_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9878657Z test_py_rref_args_user_share_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9879391Z test_py_sparse_tensors_in_container (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9880070Z test_rref_get_type_timeout_blocking (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9880753Z test_rref_get_type_timeout_non_blocking (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9881383Z test_rref_proxy_timeout (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9882024Z test_self_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9882707Z test_self_remote_rref_as_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9883407Z test_self_remote_rref_as_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9884101Z test_self_remote_rref_as_self_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9884828Z test_self_remote_rref_as_self_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9885500Z test_send_to_rank_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9886134Z test_set_and_get_num_worker_threads (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9886796Z test_stress_heavy_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9887501Z test_tensorpipe_options_throw_on_timedelta_timeout (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9888233Z test_tensorpipe_set_default_timeout (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9888886Z test_wait_all_workers_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9889548Z test_wait_all_workers_twice_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9890389Z test_world_size_one_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2023-01-11T22:03:09.9891792Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_create_remote_module_from_module_rref>, <__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_send_remote_module_over_the_wire>, <__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_send_remote_module_over_the_wire_script_not_supported>]>
2023-01-11T22:03:09.9893169Z test_create_remote_module_from_module_rref (__main__.TensorPipeThreeWorkersRemoteModuleTest)
2023-01-11T22:03:09.9893901Z test_send_remote_module_over_the_wire (__main__.TensorPipeThreeWorkersRemoteModuleTest)
2023-01-11T22:03:09.9894680Z test_send_remote_module_over_the_wire_script_not_supported (__main__.TensorPipeThreeWorkersRemoteModuleTest)
2023-01-11T22:03:09.9895770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9896415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9897264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9897936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9898585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpltqrh5nf
2023-01-11T22:03:09.9899372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpltqrh5nf/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9900291Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:09.9900698Z 
2023-01-11T22:03:09.9900841Z Running tests...
2023-01-11T22:03:09.9901401Z ----------------------------------------------------------------------
2023-01-11T22:03:09.9903203Z   test_ddp_comparison (__main__.TensorPipeDdpComparisonTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/69662 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.317s)
2023-01-11T22:03:09.9904107Z 
2023-01-11T22:03:09.9904487Z ----------------------------------------------------------------------
2023-01-11T22:03:09.9904930Z Ran 1 test in 0.317s
2023-01-11T22:03:09.9905140Z 
2023-01-11T22:03:09.9905265Z OK (skipped=1)
2023-01-11T22:03:09.9905467Z 
2023-01-11T22:03:09.9905633Z Generating XML reports...
2023-01-11T22:03:09.9906574Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20230111212834.xml
2023-01-11T22:03:09.9907648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9908266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9909104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9909785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9910443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp524q8xfk
2023-01-11T22:03:09.9911214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp524q8xfk/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9912104Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:09.9912514Z 
2023-01-11T22:03:09.9912656Z Running tests...
2023-01-11T22:03:09.9913198Z ----------------------------------------------------------------------
2023-01-11T22:03:09.9914608Z   test_ddp_comparison_uneven_inputs (__main__.TensorPipeDdpComparisonTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/72891 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.317s)
2023-01-11T22:03:09.9915548Z 
2023-01-11T22:03:09.9915862Z ----------------------------------------------------------------------
2023-01-11T22:03:09.9916214Z Ran 1 test in 0.317s
2023-01-11T22:03:09.9916381Z 
2023-01-11T22:03:09.9916465Z OK (skipped=1)
2023-01-11T22:03:09.9916631Z 
2023-01-11T22:03:09.9916756Z Generating XML reports...
2023-01-11T22:03:09.9917534Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20230111212836.xml
2023-01-11T22:03:09.9918371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9918933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9919610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9920132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9920633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31yfo4ks
2023-01-11T22:03:09.9921248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31yfo4ks/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9921976Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:09.9922298Z 
2023-01-11T22:03:09.9922403Z Running tests...
2023-01-11T22:03:09.9922858Z ----------------------------------------------------------------------
2023-01-11T22:03:09.9923483Z   test_ddp_dist_autograd_local_vs_remote (__main__.TensorPipeDdpComparisonTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3370
2023-01-11T22:03:09.9924098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3371
2023-01-11T22:03:09.9924599Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3372
2023-01-11T22:03:09.9925109Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3373
2023-01-11T22:03:09.9925822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9926320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9926968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9927475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9927995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8m219iwv
2023-01-11T22:03:09.9928573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8m219iwv/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9929322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9929823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9930479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9931001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9931515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6m59uo_h
2023-01-11T22:03:09.9932132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6m59uo_h/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9932856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9933330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9933939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9934508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9935053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7tul3pd
2023-01-11T22:03:09.9935695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7tul3pd/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9936507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9937015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9937671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9938195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9938706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdz6mdwn
2023-01-11T22:03:09.9939311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdz6mdwn/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9939886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:09.9940395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:09.9940934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:09.9941458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:09.9942171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:09.9942723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:09.9943540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:09.9944119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:09.9944689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:09.9945406Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:09.9946105Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:09.9946887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:09.9947501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:09.9948058Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:09.9948590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:09.9949159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:09.9949703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:09.9950239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:09.9950763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:09.9951294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:09.9951688Z ok (2.744s)
2023-01-11T22:03:09.9951841Z 
2023-01-11T22:03:09.9952170Z ----------------------------------------------------------------------
2023-01-11T22:03:09.9952526Z Ran 1 test in 2.745s
2023-01-11T22:03:09.9952690Z 
2023-01-11T22:03:09.9952773Z OK
2023-01-11T22:03:09.9952909Z 
2023-01-11T22:03:09.9953031Z Generating XML reports...
2023-01-11T22:03:09.9953744Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20230111212838.xml
2023-01-11T22:03:09.9954850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9955355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9956030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9956536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9957022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpduqklj2d
2023-01-11T22:03:09.9957593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpduqklj2d/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9958315Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:09.9958625Z 
2023-01-11T22:03:09.9958726Z Running tests...
2023-01-11T22:03:09.9959245Z ----------------------------------------------------------------------
2023-01-11T22:03:09.9959844Z   test_ddp_dist_autograd_sparse_grads (__main__.TensorPipeDdpComparisonTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3625
2023-01-11T22:03:09.9960487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3626
2023-01-11T22:03:09.9960986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3627
2023-01-11T22:03:09.9961463Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3628
2023-01-11T22:03:09.9962191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9962676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9963325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9963858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9964406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvs9vlaz7
2023-01-11T22:03:09.9965028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvs9vlaz7/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9965768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9966240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9966942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9967560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9968096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp04e7qmpe
2023-01-11T22:03:09.9968698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp04e7qmpe/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9969429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9969954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9970649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9971200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9971730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpml7aoh7l
2023-01-11T22:03:09.9972326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpml7aoh7l/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9973093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9973594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9974366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9974935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9975445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7n3owqmd
2023-01-11T22:03:09.9976062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7n3owqmd/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9976626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:09.9977139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:09.9977650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:09.9978179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:09.9978727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:09.9979300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:09.9979846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:09.9980411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:09.9981168Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:09.9982459Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:09.9983069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:09.9983592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:09.9983983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:09.9984344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:09.9984700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:09.9985043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:09.9985297Z ok (2.841s)
2023-01-11T22:03:09.9985401Z 
2023-01-11T22:03:09.9985609Z ----------------------------------------------------------------------
2023-01-11T22:03:09.9985853Z Ran 1 test in 2.841s
2023-01-11T22:03:09.9985953Z 
2023-01-11T22:03:09.9986014Z OK
2023-01-11T22:03:09.9986104Z 
2023-01-11T22:03:09.9986189Z Generating XML reports...
2023-01-11T22:03:09.9986686Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20230111212843.xml
2023-01-11T22:03:09.9987248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9987580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9988013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9988358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9988690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfynzt7m8
2023-01-11T22:03:09.9989093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfynzt7m8/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9989560Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:09.9989765Z 
2023-01-11T22:03:09.9989841Z Running tests...
2023-01-11T22:03:09.9990288Z ----------------------------------------------------------------------
2023-01-11T22:03:09.9990761Z   test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3892
2023-01-11T22:03:09.9991192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3893
2023-01-11T22:03:09.9991505Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3894
2023-01-11T22:03:09.9991827Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3895
2023-01-11T22:03:09.9992150Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 3896
2023-01-11T22:03:09.9992473Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 3897
2023-01-11T22:03:09.9992916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9993249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9993682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9994016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9994359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswfcrxdo
2023-01-11T22:03:09.9994762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswfcrxdo/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9995251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9995569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9995994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9996338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9996668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjwd8hbj
2023-01-11T22:03:09.9997072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjwd8hbj/_remote_module_non_scriptable.py
2023-01-11T22:03:09.9997449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:09.9997907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:09.9998223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:09.9998651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:09.9999061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:09.9999407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5rgpdq2g
2023-01-11T22:03:09.9999799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5rgpdq2g/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0000178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T22:03:10.0000646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0000961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0001392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0001736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0002077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpupgzopvq
2023-01-11T22:03:10.0002469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpupgzopvq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0002957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0003326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0003782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0004126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0004465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1nramng
2023-01-11T22:03:10.0004866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1nramng/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0005342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0005668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0006094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0006438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0006769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2xdryyj6
2023-01-11T22:03:10.0007167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2xdryyj6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0007543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T22:03:10.0007876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0008222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0008569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0009024Z 2023-01-11 21:28:50,841 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0...
2023-01-11T22:03:10.0009535Z 2023-01-11 21:28:50,841 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0010021Z 2023-01-11 21:28:50,851 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1...
2023-01-11T22:03:10.0010470Z 2023-01-11 21:28:50,851 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2...
2023-01-11T22:03:10.0010977Z 2023-01-11 21:28:50,851 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0011507Z 2023-01-11 21:28:50,851 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0011991Z 2023-01-11 21:28:50,871 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running.
2023-01-11T22:03:10.0012444Z 2023-01-11 21:28:50,871 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3...
2023-01-11T22:03:10.0012948Z 2023-01-11 21:28:50,871 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0013427Z 2023-01-11 21:28:50,872 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process...
2023-01-11T22:03:10.0013795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0014162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0014529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0014883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0015240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T22:03:10.0015643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T22:03:10.0016153Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0016672Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0017158Z 2023-01-11 21:28:50,915 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master
2023-01-11T22:03:10.0017524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T22:03:10.0018004Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0018515Z 2023-01-11 21:28:50,916 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
2023-01-11T22:03:10.0018899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T22:03:10.0019384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0019881Z 2023-01-11 21:28:50,924 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
2023-01-11T22:03:10.0020375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0020879Z 2023-01-11 21:28:50,924 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
2023-01-11T22:03:10.0021376Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0022093Z 2023-01-11 21:28:50,924 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
2023-01-11T22:03:10.0022572Z 2023-01-11 21:28:50,949 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3
2023-01-11T22:03:10.0023018Z 2023-01-11 21:28:50,949 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3
2023-01-11T22:03:10.0023389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:03:10.0023746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:03:10.0024118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T22:03:10.0024487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T22:03:10.0024975Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0025484Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0025977Z 2023-01-11 21:28:51,130 ddp_under_dist_autograd_test.py:153 INFO p:process 2 t:Dummy-2: Use DDP for the second local net.
2023-01-11T22:03:10.0026442Z 2023-01-11 21:28:51,130 ddp_under_dist_autograd_test.py:153 INFO p:process 3 t:Dummy-2: Use DDP for the second local net.
2023-01-11T22:03:10.0026936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0027439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0027930Z 2023-01-11 21:28:51,134 ddp_under_dist_autograd_test.py:153 INFO p:process 1 t:Dummy-2: Use DDP for the second local net.
2023-01-11T22:03:10.0028386Z 2023-01-11 21:28:51,134 ddp_under_dist_autograd_test.py:153 INFO p:process 0 t:Dummy-2: Use DDP for the second local net.
2023-01-11T22:03:10.0028992Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0029497Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0030136Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:644: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2023-01-11T22:03:10.0030581Z   "The `check_reduction` argument in `DistributedDataParallel` "
2023-01-11T22:03:10.0031041Z 2023-01-11 21:28:51,140 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0031651Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:644: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2023-01-11T22:03:10.0032092Z   "The `check_reduction` argument in `DistributedDataParallel` "
2023-01-11T22:03:10.0032538Z 2023-01-11 21:28:51,140 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0033154Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:644: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2023-01-11T22:03:10.0033574Z   "The `check_reduction` argument in `DistributedDataParallel` "
2023-01-11T22:03:10.0034022Z 2023-01-11 21:28:51,140 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0034581Z 2023-01-11 21:28:51,140 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params.
2023-01-11T22:03:10.0035188Z 2023-01-11 21:28:51,140 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params.
2023-01-11T22:03:10.0035775Z 2023-01-11 21:28:51,140 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params.
2023-01-11T22:03:10.0036431Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:644: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2023-01-11T22:03:10.0036873Z   "The `check_reduction` argument in `DistributedDataParallel` "
2023-01-11T22:03:10.0037318Z 2023-01-11 21:28:51,141 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0037851Z 2023-01-11 21:28:51,141 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params.
2023-01-11T22:03:10.0038430Z 2023-01-11 21:28:51,194 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0038748Z         [-1.,  1.],
2023-01-11T22:03:10.0039026Z         [-1., -1.],
2023-01-11T22:03:10.0039397Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0039815Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0040086Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0040345Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2023-01-11T22:03:10.0040583Z         [-16.,  16.]])}
2023-01-11T22:03:10.0041014Z 2023-01-11 21:28:51,194 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0041343Z         [ 1.,  1.],
2023-01-11T22:03:10.0041520Z         [ 1., -1.],
2023-01-11T22:03:10.0041931Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0042340Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0042590Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0042800Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2023-01-11T22:03:10.0042999Z         [ 0., 16.]])}
2023-01-11T22:03:10.0043417Z 2023-01-11 21:28:51,194 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2023-01-11T22:03:10.0043733Z         [-1., -1.],
2023-01-11T22:03:10.0043921Z         [-1.,  1.],
2023-01-11T22:03:10.0044311Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0044717Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0044981Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0045253Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2023-01-11T22:03:10.0045472Z         [ 16., -16.]])}
2023-01-11T22:03:10.0045898Z 2023-01-11 21:28:51,197 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2023-01-11T22:03:10.0046215Z         [ 1., -1.],
2023-01-11T22:03:10.0046362Z         [ 1.,  1.],
2023-01-11T22:03:10.0046747Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0047157Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0047423Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0047682Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2023-01-11T22:03:10.0047910Z         [  0., -16.]])}
2023-01-11T22:03:10.0048183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:10.0048534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:10.0048886Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:10.0049238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:10.0049751Z 2023-01-11 21:28:51,222 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2023-01-11T22:03:10.0050057Z         [ 1., -1.],
2023-01-11T22:03:10.0050219Z         [ 1.,  1.],
2023-01-11T22:03:10.0050606Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0051011Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0051274Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0051539Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2023-01-11T22:03:10.0051768Z         [  0., -16.]])}
2023-01-11T22:03:10.0052177Z 2023-01-11 21:28:51,222 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0052464Z         [ 1.,  1.],
2023-01-11T22:03:10.0052651Z         [ 1., -1.],
2023-01-11T22:03:10.0053018Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0053425Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0053687Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0053881Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2023-01-11T22:03:10.0054116Z         [ 0., 16.]])}
2023-01-11T22:03:10.0054580Z 2023-01-11 21:28:51,223 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2023-01-11T22:03:10.0054901Z         [-1., -1.],
2023-01-11T22:03:10.0055081Z         [-1.,  1.],
2023-01-11T22:03:10.0055467Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0055882Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0056128Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0056393Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2023-01-11T22:03:10.0056626Z         [ 16., -16.]])}
2023-01-11T22:03:10.0057055Z 2023-01-11 21:28:51,225 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0057364Z         [-1.,  1.],
2023-01-11T22:03:10.0057555Z         [-1., -1.],
2023-01-11T22:03:10.0057932Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0058328Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0058591Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0058873Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2023-01-11T22:03:10.0059092Z         [-16.,  16.]])}
2023-01-11T22:03:10.0059520Z 2023-01-11 21:28:51,248 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2023-01-11T22:03:10.0059844Z         [-1., -1.],
2023-01-11T22:03:10.0060036Z         [-1.,  1.],
2023-01-11T22:03:10.0060412Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0060826Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0061086Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0061339Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2023-01-11T22:03:10.0061572Z         [ 16., -16.]])}
2023-01-11T22:03:10.0062203Z 2023-01-11 21:28:51,248 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0062495Z         [ 1.,  1.],
2023-01-11T22:03:10.0062678Z         [ 1., -1.],
2023-01-11T22:03:10.0063058Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0063469Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0063718Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0063934Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2023-01-11T22:03:10.0064139Z         [ 0., 16.]])}
2023-01-11T22:03:10.0064558Z 2023-01-11 21:28:51,248 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2023-01-11T22:03:10.0064878Z         [ 1., -1.],
2023-01-11T22:03:10.0065039Z         [ 1.,  1.],
2023-01-11T22:03:10.0065428Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0065831Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0066092Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0066359Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2023-01-11T22:03:10.0066580Z         [  0., -16.]])}
2023-01-11T22:03:10.0067010Z 2023-01-11 21:28:51,248 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0067412Z         [-1.,  1.],
2023-01-11T22:03:10.0067591Z         [-1., -1.],
2023-01-11T22:03:10.0068042Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0068457Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0068724Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0068977Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2023-01-11T22:03:10.0069209Z         [-16.,  16.]])}
2023-01-11T22:03:10.0069578Z 2023-01-11 21:28:51,256 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0...
2023-01-11T22:03:10.0070024Z 2023-01-11 21:28:51,257 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1...
2023-01-11T22:03:10.0070475Z 2023-01-11 21:28:51,310 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2...
2023-01-11T22:03:10.0070926Z 2023-01-11 21:28:51,311 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3...
2023-01-11T22:03:10.0071373Z 2023-01-11 21:28:51,351 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker.
2023-01-11T22:03:10.0071612Z ok (4.055s)
2023-01-11T22:03:10.0071715Z 
2023-01-11T22:03:10.0071918Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0072165Z Ran 1 test in 4.055s
2023-01-11T22:03:10.0072279Z 
2023-01-11T22:03:10.0072327Z OK
2023-01-11T22:03:10.0072418Z 
2023-01-11T22:03:10.0072503Z Generating XML reports...
2023-01-11T22:03:10.0073027Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20230111212847.xml
2023-01-11T22:03:10.0073614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0073965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0074400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0074746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0075090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2h27wv6
2023-01-11T22:03:10.0075475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2h27wv6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0075941Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0076150Z 
2023-01-11T22:03:10.0076222Z Running tests...
2023-01-11T22:03:10.0076512Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0076939Z   test_backward_ddp_outside (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4306
2023-01-11T22:03:10.0077363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4307
2023-01-11T22:03:10.0077696Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4308
2023-01-11T22:03:10.0078011Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4309
2023-01-11T22:03:10.0078335Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 4310
2023-01-11T22:03:10.0078665Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 4311
2023-01-11T22:03:10.0079184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0079522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0079952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0080344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0080708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptvdpsdo
2023-01-11T22:03:10.0081118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptvdpsdo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0081613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0081945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0082362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0082714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0083059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgd524kr
2023-01-11T22:03:10.0083451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgd524kr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0083949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0084280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0084716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0085050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0085396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpemr_citk
2023-01-11T22:03:10.0085797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpemr_citk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0086165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T22:03:10.0086632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0086971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0087409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0087750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0088096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnj23xsen
2023-01-11T22:03:10.0088501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnj23xsen/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0088875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0089324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0089652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0090084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0090418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0090758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26fz1vjs
2023-01-11T22:03:10.0091160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26fz1vjs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0091645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0091957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0092381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0092723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0093050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt76aukk
2023-01-11T22:03:10.0093479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt76aukk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0093889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0094238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T22:03:10.0094570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0094909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0095371Z 2023-01-11 21:28:56,904 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0...
2023-01-11T22:03:10.0095887Z 2023-01-11 21:28:56,904 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0096358Z 2023-01-11 21:28:56,914 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1...
2023-01-11T22:03:10.0096864Z 2023-01-11 21:28:56,914 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0097341Z 2023-01-11 21:28:56,924 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2...
2023-01-11T22:03:10.0097854Z 2023-01-11 21:28:56,924 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0098328Z 2023-01-11 21:28:56,944 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running.
2023-01-11T22:03:10.0098778Z 2023-01-11 21:28:56,944 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3...
2023-01-11T22:03:10.0099281Z 2023-01-11 21:28:56,944 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0099771Z 2023-01-11 21:28:56,945 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process...
2023-01-11T22:03:10.0100130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0100498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0100863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0101231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T22:03:10.0101580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0102093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T22:03:10.0102601Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0103112Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0103635Z 2023-01-11 21:28:56,998 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
2023-01-11T22:03:10.0104143Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0104656Z 2023-01-11 21:28:56,998 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
2023-01-11T22:03:10.0105113Z 2023-01-11 21:28:56,998 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master
2023-01-11T22:03:10.0105605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0106218Z 2023-01-11 21:28:56,999 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
2023-01-11T22:03:10.0106728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0107227Z 2023-01-11 21:28:56,999 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
2023-01-11T22:03:10.0107612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T22:03:10.0108105Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0108503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T22:03:10.0108940Z 2023-01-11 21:28:57,031 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3
2023-01-11T22:03:10.0109390Z 2023-01-11 21:28:57,031 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3
2023-01-11T22:03:10.0109757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:03:10.0110126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:03:10.0110478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T22:03:10.0110838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T22:03:10.0111325Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0111831Z 2023-01-11 21:28:57,217 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0112307Z 2023-01-11 21:28:57,217 ddp_under_dist_autograd_test.py:202 INFO p:process 3 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2023-01-11T22:03:10.0112806Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0113321Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0113837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0114331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0114828Z 2023-01-11 21:28:57,225 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0115306Z 2023-01-11 21:28:57,226 ddp_under_dist_autograd_test.py:202 INFO p:process 1 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2023-01-11T22:03:10.0115784Z 2023-01-11 21:28:57,226 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0116248Z 2023-01-11 21:28:57,226 ddp_under_dist_autograd_test.py:202 INFO p:process 0 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2023-01-11T22:03:10.0116744Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0117236Z 2023-01-11 21:28:57,229 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0117713Z 2023-01-11 21:28:57,229 ddp_under_dist_autograd_test.py:202 INFO p:process 2 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2023-01-11T22:03:10.0118320Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:644: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2023-01-11T22:03:10.0118796Z   "The `check_reduction` argument in `DistributedDataParallel` "
2023-01-11T22:03:10.0119408Z 2023-01-11 21:28:57,235 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2023-01-11T22:03:10.0120065Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:644: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2023-01-11T22:03:10.0120482Z   "The `check_reduction` argument in `DistributedDataParallel` "
2023-01-11T22:03:10.0121005Z 2023-01-11 21:28:57,241 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2023-01-11T22:03:10.0121660Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:644: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2023-01-11T22:03:10.0122097Z   "The `check_reduction` argument in `DistributedDataParallel` "
2023-01-11T22:03:10.0122602Z 2023-01-11 21:28:57,241 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2023-01-11T22:03:10.0123253Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:644: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2023-01-11T22:03:10.0123684Z   "The `check_reduction` argument in `DistributedDataParallel` "
2023-01-11T22:03:10.0124200Z 2023-01-11 21:28:57,246 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2023-01-11T22:03:10.0124749Z 2023-01-11 21:28:57,302 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2023-01-11T22:03:10.0125072Z         [ 1., -1.],
2023-01-11T22:03:10.0125237Z         [ 1.,  1.],
2023-01-11T22:03:10.0125630Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0126030Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0126294Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0126501Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2023-01-11T22:03:10.0126685Z         [0., 0.]])}
2023-01-11T22:03:10.0127107Z 2023-01-11 21:28:57,302 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0127423Z         [-1.,  1.],
2023-01-11T22:03:10.0127597Z         [-1., -1.],
2023-01-11T22:03:10.0127975Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0128390Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0128649Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0128841Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2023-01-11T22:03:10.0129039Z         [0., 0.]])}
2023-01-11T22:03:10.0129461Z 2023-01-11 21:28:57,302 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0129736Z         [ 1.,  1.],
2023-01-11T22:03:10.0129924Z         [ 1., -1.],
2023-01-11T22:03:10.0130300Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0130706Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0130995Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0131202Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2023-01-11T22:03:10.0131396Z         [0., 0.]])}
2023-01-11T22:03:10.0131843Z 2023-01-11 21:28:57,305 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2023-01-11T22:03:10.0132168Z         [-1., -1.],
2023-01-11T22:03:10.0132356Z         [-1.,  1.],
2023-01-11T22:03:10.0132734Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0133146Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0133407Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0133619Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2023-01-11T22:03:10.0133804Z         [0., 0.]])}
2023-01-11T22:03:10.0134071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:10.0134431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:10.0134774Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:10.0135131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:10.0135644Z 2023-01-11 21:28:57,332 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2023-01-11T22:03:10.0135959Z         [ 1., -1.],
2023-01-11T22:03:10.0136108Z         [ 1.,  1.],
2023-01-11T22:03:10.0136496Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0136908Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0137165Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0137373Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2023-01-11T22:03:10.0137570Z         [0., 0.]])}
2023-01-11T22:03:10.0137986Z 2023-01-11 21:28:57,332 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2023-01-11T22:03:10.0138312Z         [-1., -1.],
2023-01-11T22:03:10.0138507Z         [-1.,  1.],
2023-01-11T22:03:10.0138896Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0139304Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0139568Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0139778Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2023-01-11T22:03:10.0139962Z         [0., 0.]])}
2023-01-11T22:03:10.0140383Z 2023-01-11 21:28:57,333 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0140706Z         [-1.,  1.],
2023-01-11T22:03:10.0140883Z         [-1., -1.],
2023-01-11T22:03:10.0141275Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0141807Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0142120Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0142320Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2023-01-11T22:03:10.0142518Z         [0., 0.]])}
2023-01-11T22:03:10.0142940Z 2023-01-11 21:28:57,335 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0143222Z         [ 1.,  1.],
2023-01-11T22:03:10.0143415Z         [ 1., -1.],
2023-01-11T22:03:10.0143796Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0144313Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0144570Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0144782Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2023-01-11T22:03:10.0144981Z         [0., 0.]])}
2023-01-11T22:03:10.0145390Z 2023-01-11 21:28:57,354 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2023-01-11T22:03:10.0145716Z         [-1., -1.],
2023-01-11T22:03:10.0145910Z         [-1.,  1.],
2023-01-11T22:03:10.0146282Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0146699Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0146802Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0146911Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2023-01-11T22:03:10.0146977Z         [0., 0.]])}
2023-01-11T22:03:10.0147306Z 2023-01-11 21:28:57,354 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0147399Z         [-1.,  1.],
2023-01-11T22:03:10.0147479Z         [-1., -1.],
2023-01-11T22:03:10.0147760Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0147974Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0148072Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0148179Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2023-01-11T22:03:10.0148243Z         [0., 0.]])}
2023-01-11T22:03:10.0148564Z 2023-01-11 21:28:57,354 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0148618Z         [ 1.,  1.],
2023-01-11T22:03:10.0148710Z         [ 1., -1.],
2023-01-11T22:03:10.0148991Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0149204Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0149303Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0149409Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2023-01-11T22:03:10.0149481Z         [0., 0.]])}
2023-01-11T22:03:10.0149802Z 2023-01-11 21:28:57,357 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2023-01-11T22:03:10.0149883Z         [ 1., -1.],
2023-01-11T22:03:10.0149950Z         [ 1.,  1.],
2023-01-11T22:03:10.0150241Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0150460Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0150558Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0150665Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2023-01-11T22:03:10.0150730Z         [0., 0.]])}
2023-01-11T22:03:10.0150993Z 2023-01-11 21:28:57,364 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0...
2023-01-11T22:03:10.0151235Z 2023-01-11 21:28:57,422 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1...
2023-01-11T22:03:10.0151503Z 2023-01-11 21:28:57,472 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2...
2023-01-11T22:03:10.0151757Z 2023-01-11 21:28:57,473 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3...
2023-01-11T22:03:10.0152050Z 2023-01-11 21:28:57,513 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker.
2023-01-11T22:03:10.0152116Z ok (4.257s)
2023-01-11T22:03:10.0152124Z 
2023-01-11T22:03:10.0152379Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0152457Z Ran 1 test in 4.258s
2023-01-11T22:03:10.0152463Z 
2023-01-11T22:03:10.0152524Z OK
2023-01-11T22:03:10.0152529Z 
2023-01-11T22:03:10.0152611Z Generating XML reports...
2023-01-11T22:03:10.0153005Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20230111212853.xml
2023-01-11T22:03:10.0153284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0153408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0153691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0153831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0154019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdanc3fnh
2023-01-11T22:03:10.0154218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdanc3fnh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0154477Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0154483Z 
2023-01-11T22:03:10.0154542Z Running tests...
2023-01-11T22:03:10.0154744Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0155020Z   test_backward_ddp_outside_uneven_inputs (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4717
2023-01-11T22:03:10.0155177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4718
2023-01-11T22:03:10.0155332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4719
2023-01-11T22:03:10.0155483Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4720
2023-01-11T22:03:10.0155637Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 4721
2023-01-11T22:03:10.0155786Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 4722
2023-01-11T22:03:10.0156060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0156170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0156451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0156586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0156773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpllrfqo5m
2023-01-11T22:03:10.0156973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpllrfqo5m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0157250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0157374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0157656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0157778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0157962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoosv1zk5
2023-01-11T22:03:10.0158159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoosv1zk5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0158430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0158587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0158976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0159116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0159302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0qeb5ti
2023-01-11T22:03:10.0159499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0qeb5ti/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0159763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0159886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0160169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0160306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0160495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9kde4j5
2023-01-11T22:03:10.0160691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9kde4j5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0160859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T22:03:10.0161022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0161296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0161406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0161688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0161822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0162010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpowidrmhd
2023-01-11T22:03:10.0162210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpowidrmhd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0162482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0162604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0162884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0163006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0163191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyi0hup6m
2023-01-11T22:03:10.0163384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyi0hup6m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0163550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0163709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0163870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0164036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T22:03:10.0164302Z 2023-01-11 21:29:03,062 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0...
2023-01-11T22:03:10.0164622Z 2023-01-11 21:29:03,062 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0164871Z 2023-01-11 21:29:03,072 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1...
2023-01-11T22:03:10.0165126Z 2023-01-11 21:29:03,072 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2...
2023-01-11T22:03:10.0165510Z 2023-01-11 21:29:03,072 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0165834Z 2023-01-11 21:29:03,072 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0166103Z 2023-01-11 21:29:03,092 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process...
2023-01-11T22:03:10.0166363Z 2023-01-11 21:29:03,092 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3...
2023-01-11T22:03:10.0166678Z 2023-01-11 21:29:03,092 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0166942Z 2023-01-11 21:29:03,093 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running.
2023-01-11T22:03:10.0167127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0167306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0167468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T22:03:10.0167641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0167811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0167982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T22:03:10.0168280Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0168571Z 2023-01-11 21:29:03,136 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
2023-01-11T22:03:10.0168873Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0169165Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0169431Z 2023-01-11 21:29:03,137 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master
2023-01-11T22:03:10.0169762Z 2023-01-11 21:29:03,137 ddp_under_dist_autograd_test.py:396 INFO p:process 5 t:MainThread: Running DDP + RPC test with simulating uneven inputs across trainers.
2023-01-11T22:03:10.0169924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T22:03:10.0170218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0170514Z 2023-01-11 21:29:03,137 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
2023-01-11T22:03:10.0170687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T22:03:10.0170983Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0171270Z 2023-01-11 21:29:03,139 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
2023-01-11T22:03:10.0171563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0171854Z 2023-01-11 21:29:03,139 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
2023-01-11T22:03:10.0172112Z 2023-01-11 21:29:03,187 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-2: Initing RemoteNet with 5 3
2023-01-11T22:03:10.0172397Z 2023-01-11 21:29:03,188 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-3: Initing RemoteEM with 2 3
2023-01-11T22:03:10.0172598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:03:10.0172773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T22:03:10.0172944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T22:03:10.0173116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:03:10.0173410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0173706Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0173987Z 2023-01-11 21:29:03,299 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0174270Z 2023-01-11 21:29:03,299 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0174553Z 2023-01-11 21:29:03,300 ddp_under_dist_autograd_test.py:202 INFO p:process 1 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2023-01-11T22:03:10.0174818Z 2023-01-11 21:29:03,300 ddp_under_dist_autograd_test.py:202 INFO p:process 2 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2023-01-11T22:03:10.0175117Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0175414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0175705Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0175987Z 2023-01-11 21:29:03,308 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0176262Z 2023-01-11 21:29:03,309 ddp_under_dist_autograd_test.py:202 INFO p:process 0 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2023-01-11T22:03:10.0176552Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T22:03:10.0176831Z 2023-01-11 21:29:03,311 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0177108Z 2023-01-11 21:29:03,311 ddp_under_dist_autograd_test.py:202 INFO p:process 3 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2023-01-11T22:03:10.0177520Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:644: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2023-01-11T22:03:10.0177651Z   "The `check_reduction` argument in `DistributedDataParallel` "
2023-01-11T22:03:10.0178007Z 2023-01-11 21:29:03,315 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2023-01-11T22:03:10.0178420Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:644: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2023-01-11T22:03:10.0178557Z   "The `check_reduction` argument in `DistributedDataParallel` "
2023-01-11T22:03:10.0178908Z 2023-01-11 21:29:03,316 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2023-01-11T22:03:10.0179178Z 2023-01-11 21:29:03,317 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-3: Trainer reduced input patches from 2
2023-01-11T22:03:10.0179312Z                     to 1 to simulate uneven inputs.
2023-01-11T22:03:10.0179750Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:644: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2023-01-11T22:03:10.0179888Z   "The `check_reduction` argument in `DistributedDataParallel` "
2023-01-11T22:03:10.0180237Z 2023-01-11 21:29:03,329 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2023-01-11T22:03:10.0180630Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:644: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2023-01-11T22:03:10.0180765Z   "The `check_reduction` argument in `DistributedDataParallel` "
2023-01-11T22:03:10.0181116Z 2023-01-11 21:29:03,329 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2023-01-11T22:03:10.0181391Z 2023-01-11 21:29:03,329 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-3: Trainer reduced input patches from 2
2023-01-11T22:03:10.0181490Z                     to 1 to simulate uneven inputs.
2023-01-11T22:03:10.0181989Z 2023-01-11 21:29:03,366 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2023-01-11T22:03:10.0182094Z         [-1., -1.],
2023-01-11T22:03:10.0182189Z         [-1.,  1.],
2023-01-11T22:03:10.0182487Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0182696Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2023-01-11T22:03:10.0182795Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0182910Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2023-01-11T22:03:10.0182978Z         [ 0.,  4.]])}
2023-01-11T22:03:10.0183307Z 2023-01-11 21:29:03,366 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2023-01-11T22:03:10.0183402Z         [ 1., -1.],
2023-01-11T22:03:10.0183470Z         [ 1.,  1.],
2023-01-11T22:03:10.0183759Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0183962Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2023-01-11T22:03:10.0184063Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0184173Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2023-01-11T22:03:10.0184242Z         [ 0.,  4.]])}
2023-01-11T22:03:10.0184567Z 2023-01-11 21:29:03,368 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0184665Z         [-1.,  1.],
2023-01-11T22:03:10.0184759Z         [-1., -1.],
2023-01-11T22:03:10.0185039Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0185245Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2023-01-11T22:03:10.0185343Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0185450Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2023-01-11T22:03:10.0185518Z         [ 0.,  4.]])}
2023-01-11T22:03:10.0185841Z 2023-01-11 21:29:03,370 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0185907Z         [ 1.,  1.],
2023-01-11T22:03:10.0185999Z         [ 1., -1.],
2023-01-11T22:03:10.0186267Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0186547Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2023-01-11T22:03:10.0186682Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0186796Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2023-01-11T22:03:10.0186864Z         [ 0.,  4.]])}
2023-01-11T22:03:10.0187035Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:10.0187205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:03:10.0187538Z 2023-01-11 21:29:03,389 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0187607Z         [ 1.,  1.],
2023-01-11T22:03:10.0187687Z         [ 1., -1.],
2023-01-11T22:03:10.0187971Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0188190Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0188292Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0188453Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2023-01-11T22:03:10.0188548Z         [-2.,  2.]])}
2023-01-11T22:03:10.0188872Z 2023-01-11 21:29:03,393 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0188952Z         [-1.,  1.],
2023-01-11T22:03:10.0189046Z         [-1., -1.],
2023-01-11T22:03:10.0189323Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0189534Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0189634Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0189800Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2023-01-11T22:03:10.0189898Z         [-2.,  2.]])}
2023-01-11T22:03:10.0190174Z 2023-01-11 21:29:03,397 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-4: Trainer reduced input patches from 2
2023-01-11T22:03:10.0190262Z                     to 1 to simulate uneven inputs.
2023-01-11T22:03:10.0190537Z 2023-01-11 21:29:03,397 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-4: Trainer reduced input patches from 2
2023-01-11T22:03:10.0190636Z                     to 1 to simulate uneven inputs.
2023-01-11T22:03:10.0190957Z 2023-01-11 21:29:03,417 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2023-01-11T22:03:10.0191050Z         [-1., -1.],
2023-01-11T22:03:10.0191142Z         [-1.,  1.],
2023-01-11T22:03:10.0191431Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0191650Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2023-01-11T22:03:10.0191736Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0191844Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2023-01-11T22:03:10.0191910Z         [ 0.,  4.]])}
2023-01-11T22:03:10.0192232Z 2023-01-11 21:29:03,417 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0192326Z         [-1.,  1.],
2023-01-11T22:03:10.0192418Z         [-1., -1.],
2023-01-11T22:03:10.0192698Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0192913Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2023-01-11T22:03:10.0193000Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0193145Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2023-01-11T22:03:10.0193214Z         [ 0.,  4.]])}
2023-01-11T22:03:10.0193572Z 2023-01-11 21:29:03,418 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2023-01-11T22:03:10.0193670Z         [ 1., -1.],
2023-01-11T22:03:10.0193736Z         [ 1.,  1.],
2023-01-11T22:03:10.0194030Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0194249Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2023-01-11T22:03:10.0194334Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0194445Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2023-01-11T22:03:10.0194514Z         [ 0.,  4.]])}
2023-01-11T22:03:10.0194839Z 2023-01-11 21:29:03,420 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0194911Z         [ 1.,  1.],
2023-01-11T22:03:10.0195006Z         [ 1., -1.],
2023-01-11T22:03:10.0195289Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0195492Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2023-01-11T22:03:10.0195590Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0195702Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2023-01-11T22:03:10.0195771Z         [ 0.,  4.]])}
2023-01-11T22:03:10.0196101Z 2023-01-11 21:29:03,432 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0196168Z         [ 1.,  1.],
2023-01-11T22:03:10.0196263Z         [ 1., -1.],
2023-01-11T22:03:10.0196541Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0196747Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0196848Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0197014Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2023-01-11T22:03:10.0197113Z         [-2.,  2.]])}
2023-01-11T22:03:10.0197436Z 2023-01-11 21:29:03,435 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0197532Z         [-1.,  1.],
2023-01-11T22:03:10.0197626Z         [-1., -1.],
2023-01-11T22:03:10.0197907Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0198106Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0198209Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0198373Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2023-01-11T22:03:10.0198470Z         [-2.,  2.]])}
2023-01-11T22:03:10.0198747Z 2023-01-11 21:29:03,438 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-5: Trainer reduced input patches from 2
2023-01-11T22:03:10.0198942Z                     to 1 to simulate uneven inputs.
2023-01-11T22:03:10.0199225Z 2023-01-11 21:29:03,439 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-5: Trainer reduced input patches from 2
2023-01-11T22:03:10.0199327Z                     to 1 to simulate uneven inputs.
2023-01-11T22:03:10.0199635Z 2023-01-11 21:29:03,455 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2023-01-11T22:03:10.0199736Z         [-1., -1.],
2023-01-11T22:03:10.0199829Z         [-1.,  1.],
2023-01-11T22:03:10.0200121Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0200382Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2023-01-11T22:03:10.0200528Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0200640Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2023-01-11T22:03:10.0200707Z         [ 0.,  4.]])}
2023-01-11T22:03:10.0201019Z 2023-01-11 21:29:03,455 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0201115Z         [-1.,  1.],
2023-01-11T22:03:10.0201210Z         [-1., -1.],
2023-01-11T22:03:10.0201493Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0201710Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2023-01-11T22:03:10.0201811Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0201924Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2023-01-11T22:03:10.0201978Z         [ 0.,  4.]])}
2023-01-11T22:03:10.0202306Z 2023-01-11 21:29:03,458 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0202373Z         [ 1.,  1.],
2023-01-11T22:03:10.0202467Z         [ 1., -1.],
2023-01-11T22:03:10.0202745Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0202964Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2023-01-11T22:03:10.0203061Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0203168Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2023-01-11T22:03:10.0203221Z         [ 0.,  4.]])}
2023-01-11T22:03:10.0203548Z 2023-01-11 21:29:03,461 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2023-01-11T22:03:10.0203648Z         [ 1., -1.],
2023-01-11T22:03:10.0203712Z         [ 1.,  1.],
2023-01-11T22:03:10.0204006Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0204220Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2023-01-11T22:03:10.0204315Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0204422Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2023-01-11T22:03:10.0204475Z         [ 0.,  4.]])}
2023-01-11T22:03:10.0204894Z 2023-01-11 21:29:03,470 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0204995Z         [ 1.,  1.],
2023-01-11T22:03:10.0205122Z         [ 1., -1.],
2023-01-11T22:03:10.0205560Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0205782Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0205912Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0206104Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2023-01-11T22:03:10.0206240Z         [-2.,  2.]])}
2023-01-11T22:03:10.0206596Z 2023-01-11 21:29:03,470 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0206722Z         [-1.,  1.],
2023-01-11T22:03:10.0206848Z         [-1., -1.],
2023-01-11T22:03:10.0207118Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0207365Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2023-01-11T22:03:10.0207572Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0207771Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2023-01-11T22:03:10.0207911Z         [-2.,  2.]])}
2023-01-11T22:03:10.0208240Z 2023-01-11 21:29:03,475 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0...
2023-01-11T22:03:10.0208540Z 2023-01-11 21:29:03,476 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1...
2023-01-11T22:03:10.0208828Z 2023-01-11 21:29:03,534 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2...
2023-01-11T22:03:10.0209110Z 2023-01-11 21:29:03,535 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3...
2023-01-11T22:03:10.0209353Z 2023-01-11 21:29:03,536 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker.
2023-01-11T22:03:10.0209684Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpg9kde4j5'>
2023-01-11T22:03:10.0209861Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T22:03:10.0210186Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpowidrmhd'>
2023-01-11T22:03:10.0210328Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T22:03:10.0210644Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpv0qeb5ti'>
2023-01-11T22:03:10.0210784Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T22:03:10.0211108Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpoosv1zk5'>
2023-01-11T22:03:10.0211207Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T22:03:10.0211304Z ok (4.051s)
2023-01-11T22:03:10.0211311Z 
2023-01-11T22:03:10.0211553Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0211678Z Ran 1 test in 4.051s
2023-01-11T22:03:10.0211687Z 
2023-01-11T22:03:10.0211777Z OK
2023-01-11T22:03:10.0211783Z 
2023-01-11T22:03:10.0211896Z Generating XML reports...
2023-01-11T22:03:10.0212332Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20230111212900.xml
2023-01-11T22:03:10.0212648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0212805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0213075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0213253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0213480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfcrn8dt
2023-01-11T22:03:10.0213737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfcrn8dt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0214025Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0214031Z 
2023-01-11T22:03:10.0214135Z Running tests...
2023-01-11T22:03:10.0214367Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0214657Z   test_backward_no_ddp (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5137
2023-01-11T22:03:10.0214800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5138
2023-01-11T22:03:10.0214994Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5139
2023-01-11T22:03:10.0215178Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5140
2023-01-11T22:03:10.0215363Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 5141
2023-01-11T22:03:10.0215564Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 5142
2023-01-11T22:03:10.0215918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0216107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0216435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0216601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0216776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0x0al9po
2023-01-11T22:03:10.0217006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0x0al9po/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0217309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0217466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0217804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0217971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0218196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87cv97fv
2023-01-11T22:03:10.0218419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87cv97fv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0218677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0218831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0219141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0219305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0219518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5hizk_5
2023-01-11T22:03:10.0219764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5hizk_5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0220081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0220237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0220547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0220668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0220882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqd1no6k
2023-01-11T22:03:10.0221140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqd1no6k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0221334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0221530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0222083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0222243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0222565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0222689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0222906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb51462pq
2023-01-11T22:03:10.0223129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb51462pq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0223431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0223654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0224014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0224204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0224420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqzu2kj8
2023-01-11T22:03:10.0224649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqzu2kj8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0224801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T22:03:10.0225027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0225216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T22:03:10.0225414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0225721Z 2023-01-11 21:29:09,160 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0...
2023-01-11T22:03:10.0226079Z 2023-01-11 21:29:09,160 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0226396Z 2023-01-11 21:29:09,180 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1...
2023-01-11T22:03:10.0226749Z 2023-01-11 21:29:09,180 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0227044Z 2023-01-11 21:29:09,190 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2...
2023-01-11T22:03:10.0227353Z 2023-01-11 21:29:09,190 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0227653Z 2023-01-11 21:29:09,200 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process...
2023-01-11T22:03:10.0227948Z 2023-01-11 21:29:09,200 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3...
2023-01-11T22:03:10.0228294Z 2023-01-11 21:29:09,200 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
2023-01-11T22:03:10.0228590Z 2023-01-11 21:29:09,200 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running.
2023-01-11T22:03:10.0228797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0229023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0229229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0229432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0229674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T22:03:10.0229961Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0230162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T22:03:10.0230482Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0230808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0231135Z 2023-01-11 21:29:09,245 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
2023-01-11T22:03:10.0231491Z 2023-01-11 21:29:09,245 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master
2023-01-11T22:03:10.0231852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0314083Z 2023-01-11 21:29:09,246 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
2023-01-11T22:03:10.0314624Z 2023-01-11 21:29:09,248 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0314986Z 2023-01-11 21:29:09,248 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params.
2023-01-11T22:03:10.0315289Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0315588Z 2023-01-11 21:29:09,254 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
2023-01-11T22:03:10.0315888Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T22:03:10.0316177Z 2023-01-11 21:29:09,255 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
2023-01-11T22:03:10.0316448Z 2023-01-11 21:29:09,288 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0316794Z 2023-01-11 21:29:09,288 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params.
2023-01-11T22:03:10.0317046Z 2023-01-11 21:29:09,322 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3
2023-01-11T22:03:10.0317298Z 2023-01-11 21:29:09,334 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3
2023-01-11T22:03:10.0317619Z 2023-01-11 21:29:09,362 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2023-01-11T22:03:10.0317711Z         [ 1., -1.],
2023-01-11T22:03:10.0317763Z         [ 1.,  1.],
2023-01-11T22:03:10.0318056Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0318276Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing:
2023-01-11T22:03:10.0318369Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0318530Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2023-01-11T22:03:10.0318625Z         [  0., -16.]])}
2023-01-11T22:03:10.0319022Z 2023-01-11 21:29:09,363 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2023-01-11T22:03:10.0319117Z         [-1., -1.],
2023-01-11T22:03:10.0319197Z         [-1.,  1.],
2023-01-11T22:03:10.0319482Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0319698Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[  0., -28., -28.]]), Parameter containing:
2023-01-11T22:03:10.0319790Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0319950Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2023-01-11T22:03:10.0320044Z         [ 16., -16.]])}
2023-01-11T22:03:10.0320321Z 2023-01-11 21:29:09,371 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0320668Z 2023-01-11 21:29:09,372 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params.
2023-01-11T22:03:10.0321135Z 2023-01-11 21:29:09,375 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters.
2023-01-11T22:03:10.0321528Z 2023-01-11 21:29:09,375 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params.
2023-01-11T22:03:10.0321850Z 2023-01-11 21:29:09,396 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0321910Z         [ 1.,  1.],
2023-01-11T22:03:10.0321997Z         [ 1., -1.],
2023-01-11T22:03:10.0322276Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0322489Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing:
2023-01-11T22:03:10.0322581Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0322690Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2023-01-11T22:03:10.0322744Z         [ 0., 16.]])}
2023-01-11T22:03:10.0323062Z 2023-01-11 21:29:09,402 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0323148Z         [-1.,  1.],
2023-01-11T22:03:10.0323233Z         [-1., -1.],
2023-01-11T22:03:10.0323509Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0323721Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing:
2023-01-11T22:03:10.0323811Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0323971Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2023-01-11T22:03:10.0324057Z         [-16.,  16.]])}
2023-01-11T22:03:10.0324380Z 2023-01-11 21:29:09,420 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2023-01-11T22:03:10.0324472Z         [-1., -1.],
2023-01-11T22:03:10.0324558Z         [-1.,  1.],
2023-01-11T22:03:10.0324844Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0325062Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[  0., -28., -28.]]), Parameter containing:
2023-01-11T22:03:10.0325153Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0325309Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2023-01-11T22:03:10.0325396Z         [ 16., -16.]])}
2023-01-11T22:03:10.0325712Z 2023-01-11 21:29:09,421 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0325799Z         [-1.,  1.],
2023-01-11T22:03:10.0325885Z         [-1., -1.],
2023-01-11T22:03:10.0326162Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0326375Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing:
2023-01-11T22:03:10.0326470Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0326622Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2023-01-11T22:03:10.0326714Z         [-16.,  16.]])}
2023-01-11T22:03:10.0327030Z 2023-01-11 21:29:09,426 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2023-01-11T22:03:10.0327119Z         [ 1., -1.],
2023-01-11T22:03:10.0327178Z         [ 1.,  1.],
2023-01-11T22:03:10.0327466Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0327683Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing:
2023-01-11T22:03:10.0327808Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0327961Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2023-01-11T22:03:10.0328053Z         [  0., -16.]])}
2023-01-11T22:03:10.0328441Z 2023-01-11 21:29:09,430 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0328502Z         [ 1.,  1.],
2023-01-11T22:03:10.0328589Z         [ 1., -1.],
2023-01-11T22:03:10.0328860Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0329072Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing:
2023-01-11T22:03:10.0329163Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0329262Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2023-01-11T22:03:10.0329321Z         [ 0., 16.]])}
2023-01-11T22:03:10.0329637Z 2023-01-11 21:29:09,446 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2023-01-11T22:03:10.0329727Z         [ 1., -1.],
2023-01-11T22:03:10.0329786Z         [ 1.,  1.],
2023-01-11T22:03:10.0330072Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0330288Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing:
2023-01-11T22:03:10.0330379Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0330532Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2023-01-11T22:03:10.0330624Z         [  0., -16.]])}
2023-01-11T22:03:10.0330941Z 2023-01-11 21:29:09,447 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2023-01-11T22:03:10.0330998Z         [ 1.,  1.],
2023-01-11T22:03:10.0331084Z         [ 1., -1.],
2023-01-11T22:03:10.0331362Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0331574Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing:
2023-01-11T22:03:10.0331666Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0331765Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2023-01-11T22:03:10.0331824Z         [ 0., 16.]])}
2023-01-11T22:03:10.0332144Z 2023-01-11 21:29:09,450 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2023-01-11T22:03:10.0332229Z         [-1., -1.],
2023-01-11T22:03:10.0332313Z         [-1.,  1.],
2023-01-11T22:03:10.0332596Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0332808Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[  0., -28., -28.]]), Parameter containing:
2023-01-11T22:03:10.0332897Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0333054Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2023-01-11T22:03:10.0333147Z         [ 16., -16.]])}
2023-01-11T22:03:10.0333462Z 2023-01-11 21:29:09,455 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2023-01-11T22:03:10.0333549Z         [-1.,  1.],
2023-01-11T22:03:10.0333633Z         [-1., -1.],
2023-01-11T22:03:10.0333908Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2023-01-11T22:03:10.0334116Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing:
2023-01-11T22:03:10.0334202Z tensor([[-1.,  1.],
2023-01-11T22:03:10.0334361Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2023-01-11T22:03:10.0334452Z         [-16.,  16.]])}
2023-01-11T22:03:10.0334755Z 2023-01-11 21:29:09,460 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0...
2023-01-11T22:03:10.0335035Z 2023-01-11 21:29:09,461 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1...
2023-01-11T22:03:10.0335283Z 2023-01-11 21:29:09,517 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2...
2023-01-11T22:03:10.0335526Z 2023-01-11 21:29:09,550 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3...
2023-01-11T22:03:10.0335770Z 2023-01-11 21:29:09,551 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker.
2023-01-11T22:03:10.0335825Z ok (4.047s)
2023-01-11T22:03:10.0335839Z 
2023-01-11T22:03:10.0336028Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0336098Z Ran 1 test in 4.047s
2023-01-11T22:03:10.0336103Z 
2023-01-11T22:03:10.0336156Z OK
2023-01-11T22:03:10.0336164Z 
2023-01-11T22:03:10.0336242Z Generating XML reports...
2023-01-11T22:03:10.0336638Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20230111212906.xml
2023-01-11T22:03:10.0336913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0337032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0337309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0337434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0337615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8r5hcbyf
2023-01-11T22:03:10.0337806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8r5hcbyf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0338054Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0338061Z 
2023-01-11T22:03:10.0338128Z Running tests...
2023-01-11T22:03:10.0338324Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0338461Z   test_async_dist_autograd (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:10.0338689Z This test ensures async processing for distributed autograd works ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5539
2023-01-11T22:03:10.0338841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5540
2023-01-11T22:03:10.0338982Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5541
2023-01-11T22:03:10.0339128Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5542
2023-01-11T22:03:10.0339397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0339519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0339798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0339927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0340108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpixl5ja11
2023-01-11T22:03:10.0340295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpixl5ja11/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0340554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0340673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0340948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0341076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0341290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplohwouqm
2023-01-11T22:03:10.0341508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplohwouqm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0342062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0342185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0342465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0342587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0342767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7abhclol
2023-01-11T22:03:10.0342954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7abhclol/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0343223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0343341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0343618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0343745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0343926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl44jjhlz
2023-01-11T22:03:10.0344107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl44jjhlz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0344265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0344425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0344580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0344739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0344913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0345086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0345383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0345674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0345836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0346005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0346296Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0346585Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0346647Z ok (2.737s)
2023-01-11T22:03:10.0346654Z 
2023-01-11T22:03:10.0346845Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0346914Z Ran 1 test in 2.738s
2023-01-11T22:03:10.0346919Z 
2023-01-11T22:03:10.0346971Z OK
2023-01-11T22:03:10.0346976Z 
2023-01-11T22:03:10.0347052Z Generating XML reports...
2023-01-11T22:03:10.0347416Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212912.xml
2023-01-11T22:03:10.0347683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0347864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0348141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0348311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0348496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2cdysk0
2023-01-11T22:03:10.0348686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2cdysk0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0348934Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0348940Z 
2023-01-11T22:03:10.0349006Z Running tests...
2023-01-11T22:03:10.0349192Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0349424Z   test_autograd_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5854
2023-01-11T22:03:10.0349575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5855
2023-01-11T22:03:10.0349723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5856
2023-01-11T22:03:10.0349869Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5857
2023-01-11T22:03:10.0350140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0350256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0350531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0350656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0350835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuybhd96h
2023-01-11T22:03:10.0351024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuybhd96h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0351294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0351415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0351692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0351820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0352000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubeygxkh
2023-01-11T22:03:10.0352190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubeygxkh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0352450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0352566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0352844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0352973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0353152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp215ulnyp
2023-01-11T22:03:10.0353338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp215ulnyp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0353601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0353717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0353986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0354114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0354323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphime3faq
2023-01-11T22:03:10.0354511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphime3faq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0354703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0354866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0355020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0355175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0355234Z ok (3.145s)
2023-01-11T22:03:10.0355240Z 
2023-01-11T22:03:10.0355427Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0355495Z Ran 1 test in 3.145s
2023-01-11T22:03:10.0355501Z 
2023-01-11T22:03:10.0355552Z OK
2023-01-11T22:03:10.0355558Z 
2023-01-11T22:03:10.0355636Z Generating XML reports...
2023-01-11T22:03:10.0355997Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212916.xml
2023-01-11T22:03:10.0356269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0356388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0356660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0356783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0356960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmx4bgaue
2023-01-11T22:03:10.0357149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmx4bgaue/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0357395Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0357403Z 
2023-01-11T22:03:10.0357470Z Running tests...
2023-01-11T22:03:10.0357664Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0357904Z   test_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6077
2023-01-11T22:03:10.0358051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6078
2023-01-11T22:03:10.0358196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6079
2023-01-11T22:03:10.0358336Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6080
2023-01-11T22:03:10.0358602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0358719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0359072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0359203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0359383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgyvdqo8a
2023-01-11T22:03:10.0359572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgyvdqo8a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0359841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0359957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0360226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0360353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0360528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4n9qzi6p
2023-01-11T22:03:10.0360751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4n9qzi6p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0361059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0361176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0361447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0361573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0361743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhduqfdx
2023-01-11T22:03:10.0361932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhduqfdx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0362193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0362312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0362587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0362713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0362888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptlia6u11
2023-01-11T22:03:10.0363075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptlia6u11/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0363230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0363382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0363535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0363694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0363758Z ok (2.649s)
2023-01-11T22:03:10.0363764Z 
2023-01-11T22:03:10.0363959Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0364032Z Ran 1 test in 2.649s
2023-01-11T22:03:10.0364038Z 
2023-01-11T22:03:10.0364099Z OK
2023-01-11T22:03:10.0364104Z 
2023-01-11T22:03:10.0364189Z Generating XML reports...
2023-01-11T22:03:10.0364546Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212921.xml
2023-01-11T22:03:10.0364822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0364944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0365220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0365351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0365527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphyo47lw6
2023-01-11T22:03:10.0365716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphyo47lw6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0365964Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0365969Z 
2023-01-11T22:03:10.0366035Z Running tests...
2023-01-11T22:03:10.0366215Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0366459Z   test_backward_autograd_engine_error (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6320
2023-01-11T22:03:10.0366607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6321
2023-01-11T22:03:10.0366752Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6322
2023-01-11T22:03:10.0366927Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6323
2023-01-11T22:03:10.0367272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0367391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0367669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0367792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0367973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyx1b03m7
2023-01-11T22:03:10.0368160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyx1b03m7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0368427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0368548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0368825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0368953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0369132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoot7zr3z
2023-01-11T22:03:10.0369320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoot7zr3z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0369578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0369695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0369968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0370094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0370273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnl3ma2x
2023-01-11T22:03:10.0370462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnl3ma2x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0370726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0370845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0371119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0371240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0371417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpp547jr
2023-01-11T22:03:10.0371603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpp547jr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0371763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0371921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0372077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0372233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0372294Z ok (2.745s)
2023-01-11T22:03:10.0372300Z 
2023-01-11T22:03:10.0372486Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0372553Z Ran 1 test in 2.746s
2023-01-11T22:03:10.0372558Z 
2023-01-11T22:03:10.0372613Z OK
2023-01-11T22:03:10.0372618Z 
2023-01-11T22:03:10.0372694Z Generating XML reports...
2023-01-11T22:03:10.0373055Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212926.xml
2023-01-11T22:03:10.0373350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0373495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0373769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0373895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0374066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9_by_ct
2023-01-11T22:03:10.0374251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9_by_ct/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0374497Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0374503Z 
2023-01-11T22:03:10.0374570Z Running tests...
2023-01-11T22:03:10.0374759Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0375001Z   test_backward_complex_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6563
2023-01-11T22:03:10.0375154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6564
2023-01-11T22:03:10.0375300Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6565
2023-01-11T22:03:10.0375439Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6566
2023-01-11T22:03:10.0375707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0375823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0376096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0376225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0376405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzext1o4o
2023-01-11T22:03:10.0376594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzext1o4o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0376861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0376980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0377248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0377374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0377554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdo3jgr4_
2023-01-11T22:03:10.0377743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdo3jgr4_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0378008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0378125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0378400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0378527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0378699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4k0sughj
2023-01-11T22:03:10.0378890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4k0sughj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0379156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0379273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0379551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0379710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0379916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprzkkrqlq
2023-01-11T22:03:10.0380107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprzkkrqlq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0380265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0380416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0380571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0380723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0380784Z ok (2.643s)
2023-01-11T22:03:10.0380789Z 
2023-01-11T22:03:10.0380983Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0381054Z Ran 1 test in 2.643s
2023-01-11T22:03:10.0381060Z 
2023-01-11T22:03:10.0381113Z OK
2023-01-11T22:03:10.0381118Z 
2023-01-11T22:03:10.0381196Z Generating XML reports...
2023-01-11T22:03:10.0381557Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212931.xml
2023-01-11T22:03:10.0381981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0382098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0382375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0382504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0382685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm27wd6p4
2023-01-11T22:03:10.0382879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm27wd6p4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0383131Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0383137Z 
2023-01-11T22:03:10.0383204Z Running tests...
2023-01-11T22:03:10.0383392Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0383632Z   test_backward_different_dtypes (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6806
2023-01-11T22:03:10.0383780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6807
2023-01-11T22:03:10.0383927Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6808
2023-01-11T22:03:10.0384073Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6809
2023-01-11T22:03:10.0384340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0384460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0384736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0384866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0385040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63en891i
2023-01-11T22:03:10.0385228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63en891i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0385492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0385608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0385881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0386060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0386281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr874beyi
2023-01-11T22:03:10.0386468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr874beyi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0386726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0386844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0387117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0387245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0387420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30givttq
2023-01-11T22:03:10.0387605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30givttq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0387872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0387990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0388263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0388384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0388561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7spsl5jc
2023-01-11T22:03:10.0388747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7spsl5jc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0388905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0389065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0389222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0389379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0389439Z ok (2.646s)
2023-01-11T22:03:10.0389445Z 
2023-01-11T22:03:10.0389631Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0389703Z Ran 1 test in 2.646s
2023-01-11T22:03:10.0389708Z 
2023-01-11T22:03:10.0389761Z OK
2023-01-11T22:03:10.0389767Z 
2023-01-11T22:03:10.0389850Z Generating XML reports...
2023-01-11T22:03:10.0390213Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212935.xml
2023-01-11T22:03:10.0390488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0390606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0390887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0391019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0391189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9cv0z6gi
2023-01-11T22:03:10.0391378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9cv0z6gi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0391630Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0391635Z 
2023-01-11T22:03:10.0391704Z Running tests...
2023-01-11T22:03:10.0391898Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0392143Z   test_backward_different_tensor_dims (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7049
2023-01-11T22:03:10.0392326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7050
2023-01-11T22:03:10.0392475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7051
2023-01-11T22:03:10.0392641Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7052
2023-01-11T22:03:10.0392915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0393038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0393315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0393449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0393636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkau8ps81
2023-01-11T22:03:10.0393828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkau8ps81/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0394098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0394218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0394488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0394621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0394798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4ul1g0u
2023-01-11T22:03:10.0394990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4ul1g0u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0395258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0395379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0395657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0395791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0395962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gjpvv52
2023-01-11T22:03:10.0396149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gjpvv52/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0396413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0396535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0396814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0396944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0397122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpib4tqn01
2023-01-11T22:03:10.0397319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpib4tqn01/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0397481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0397633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0397793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0397948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0398009Z ok (2.753s)
2023-01-11T22:03:10.0398015Z 
2023-01-11T22:03:10.0398212Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0398283Z Ran 1 test in 2.753s
2023-01-11T22:03:10.0398289Z 
2023-01-11T22:03:10.0398346Z OK
2023-01-11T22:03:10.0398352Z 
2023-01-11T22:03:10.0398429Z Generating XML reports...
2023-01-11T22:03:10.0398887Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212940.xml
2023-01-11T22:03:10.0399203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0399325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0399608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0399738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0399918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdof0bnpl
2023-01-11T22:03:10.0400117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdof0bnpl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0400370Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0400378Z 
2023-01-11T22:03:10.0400447Z Running tests...
2023-01-11T22:03:10.0400633Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0400874Z   test_backward_invalid_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7292
2023-01-11T22:03:10.0401026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7293
2023-01-11T22:03:10.0401175Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7294
2023-01-11T22:03:10.0401325Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7295
2023-01-11T22:03:10.0401597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0401719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0401997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0402134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0402311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp15bkpuld
2023-01-11T22:03:10.0402505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp15bkpuld/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0402780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0402898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0403174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0403312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0403495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpct2m7v86
2023-01-11T22:03:10.0403686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpct2m7v86/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0403946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0404065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0404342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0404471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0404652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2l4y191
2023-01-11T22:03:10.0404840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2l4y191/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0405107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0405259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0405578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0405703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0405881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6q60mgl
2023-01-11T22:03:10.0406073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6q60mgl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0406234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0406391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0406551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0406708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0406775Z ok (2.639s)
2023-01-11T22:03:10.0406781Z 
2023-01-11T22:03:10.0406967Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0407040Z Ran 1 test in 2.639s
2023-01-11T22:03:10.0407046Z 
2023-01-11T22:03:10.0407103Z OK
2023-01-11T22:03:10.0407109Z 
2023-01-11T22:03:10.0407187Z Generating XML reports...
2023-01-11T22:03:10.0407551Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212944.xml
2023-01-11T22:03:10.0407824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0407942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0408224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0408354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0408530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg17sw3m
2023-01-11T22:03:10.0408726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg17sw3m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0408974Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0408980Z 
2023-01-11T22:03:10.0409049Z Running tests...
2023-01-11T22:03:10.0409244Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0409979Z   test_backward_multiple_output_tensors (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77516 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.322s)
2023-01-11T22:03:10.0409988Z 
2023-01-11T22:03:10.0410185Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0410255Z Ran 1 test in 0.322s
2023-01-11T22:03:10.0410260Z 
2023-01-11T22:03:10.0410328Z OK (skipped=1)
2023-01-11T22:03:10.0410334Z 
2023-01-11T22:03:10.0410415Z Generating XML reports...
2023-01-11T22:03:10.0410772Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212949.xml
2023-01-11T22:03:10.0411041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0411160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0411442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0411572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0411789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpko93_ihh
2023-01-11T22:03:10.0411978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpko93_ihh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0412257Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0412264Z 
2023-01-11T22:03:10.0412335Z Running tests...
2023-01-11T22:03:10.0412520Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0412760Z   test_backward_multiple_roots (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7529
2023-01-11T22:03:10.0412917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7530
2023-01-11T22:03:10.0413067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7531
2023-01-11T22:03:10.0413216Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7532
2023-01-11T22:03:10.0413493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0413616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0413895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0414019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0414201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdempnu6j
2023-01-11T22:03:10.0414396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdempnu6j/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0414667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0414788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0415066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0415199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0415385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi4mofpzb
2023-01-11T22:03:10.0415655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0415765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0416042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0416173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0416358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp217nn321
2023-01-11T22:03:10.0416630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0416754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0417032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0417164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0417338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshigvkpt
2023-01-11T22:03:10.0417534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi4mofpzb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0417717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp217nn321/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0417905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshigvkpt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0418067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0418263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0418455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0418616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0418678Z ok (2.843s)
2023-01-11T22:03:10.0418684Z 
2023-01-11T22:03:10.0418872Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0418944Z Ran 1 test in 2.844s
2023-01-11T22:03:10.0418950Z 
2023-01-11T22:03:10.0419006Z OK
2023-01-11T22:03:10.0419011Z 
2023-01-11T22:03:10.0419092Z Generating XML reports...
2023-01-11T22:03:10.0419461Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212951.xml
2023-01-11T22:03:10.0419732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0419857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0420139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0420262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0420445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubk1u8rt
2023-01-11T22:03:10.0420636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubk1u8rt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0420890Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0420895Z 
2023-01-11T22:03:10.0420964Z Running tests...
2023-01-11T22:03:10.0421160Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0422065Z   test_backward_multiple_round_trips (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77453 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.320s)
2023-01-11T22:03:10.0422077Z 
2023-01-11T22:03:10.0422278Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0422352Z Ran 1 test in 0.320s
2023-01-11T22:03:10.0422356Z 
2023-01-11T22:03:10.0422427Z OK (skipped=1)
2023-01-11T22:03:10.0422432Z 
2023-01-11T22:03:10.0422504Z Generating XML reports...
2023-01-11T22:03:10.0422873Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212956.xml
2023-01-11T22:03:10.0423146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0423265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0423547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0423683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0423868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpowqq4hvt
2023-01-11T22:03:10.0424063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpowqq4hvt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0424316Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0424322Z 
2023-01-11T22:03:10.0424382Z Running tests...
2023-01-11T22:03:10.0424581Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0424825Z   test_backward_no_grad_on_tensor (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7782
2023-01-11T22:03:10.0425050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7783
2023-01-11T22:03:10.0425199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7784
2023-01-11T22:03:10.0425388Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7785
2023-01-11T22:03:10.0425665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0425784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0426064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0426187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0426369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzz_6iz8r
2023-01-11T22:03:10.0426643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0426766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0427048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0427179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0427364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpylwl48tq
2023-01-11T22:03:10.0427555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzz_6iz8r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0427733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpylwl48tq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0428003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0428125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0428400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0428533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0428716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cxpkv9y
2023-01-11T22:03:10.0428907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cxpkv9y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0429177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0429296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0429565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0429698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0429876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr60wy_x_
2023-01-11T22:03:10.0430067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr60wy_x_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0430228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0430392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0430548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0430707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0430761Z ok (2.748s)
2023-01-11T22:03:10.0430767Z 
2023-01-11T22:03:10.0430966Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0431037Z Ran 1 test in 2.749s
2023-01-11T22:03:10.0431043Z 
2023-01-11T22:03:10.0431102Z OK
2023-01-11T22:03:10.0431107Z 
2023-01-11T22:03:10.0431225Z Generating XML reports...
2023-01-11T22:03:10.0431592Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212958.xml
2023-01-11T22:03:10.0431894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0432016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0432295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0432418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0432598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpknyj_ity
2023-01-11T22:03:10.0432789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpknyj_ity/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0433043Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0433052Z 
2023-01-11T22:03:10.0433125Z Running tests...
2023-01-11T22:03:10.0433317Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0433560Z   test_backward_node_failure (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8025
2023-01-11T22:03:10.0433715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8026
2023-01-11T22:03:10.0433855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8027
2023-01-11T22:03:10.0434001Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8028
2023-01-11T22:03:10.0434277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0434396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0434674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0434812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0434998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpga4tcdbd
2023-01-11T22:03:10.0435194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpga4tcdbd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0435463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0435575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0435855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0435988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0436170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmc1yusq
2023-01-11T22:03:10.0436363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmc1yusq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0436633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0436755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0437035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0437157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0437341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9enpgw_q
2023-01-11T22:03:10.0437534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9enpgw_q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0437804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0437966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0438274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0438409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0438592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_fld_ru
2023-01-11T22:03:10.0438783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_fld_ru/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0439020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0439186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0439345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0439506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0439686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0439863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0440171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0440344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0440514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0440798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0441092Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0441382Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0441652Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0441921Z [W tensorpipe_agent.cpp:678] RPC agent for worker2 encountered error when sending response to request #2 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0442187Z [W tensorpipe_agent.cpp:939] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0442486Z [W tensorpipe_agent.cpp:915] RPC agent for worker2 encountered error when sending outgoing request #2 to worker1: ECONNREFUSED: connection refused (this error originated at tensorpipe/transport/uv/connection_impl.cc:62)
2023-01-11T22:03:10.0442750Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0443039Z [W tensorpipe_agent.cpp:915] RPC agent for worker2 encountered error when sending outgoing request #3 to worker1: ECONNREFUSED: connection refused (this error originated at tensorpipe/transport/uv/connection_impl.cc:62)
2023-01-11T22:03:10.0443300Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0443559Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0443819Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #5 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0444148Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0444404Z [W tensorpipe_agent.cpp:939] RPC agent for worker2 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0444622Z [E container.cpp:256] Could not release Dist Autograd Context on node 0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0444876Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0445087Z [E container.cpp:256] Could not release Dist Autograd Context on node 2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0445349Z [W tensorpipe_agent.cpp:915] RPC agent for worker2 encountered error when sending outgoing request #5 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0445609Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #6 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0445868Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #7 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0446160Z [W tensorpipe_agent.cpp:915] RPC agent for worker2 encountered error when sending outgoing request #6 to worker1: ECONNREFUSED: connection refused (this error originated at tensorpipe/transport/uv/connection_impl.cc:62)
2023-01-11T22:03:10.0446420Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #8 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0446674Z [W tensorpipe_agent.cpp:915] RPC agent for worker2 encountered error when sending outgoing request #7 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0446932Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #9 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0447188Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0447445Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0447503Z ok (2.762s)
2023-01-11T22:03:10.0447519Z 
2023-01-11T22:03:10.0447710Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0447781Z Ran 1 test in 2.763s
2023-01-11T22:03:10.0447787Z 
2023-01-11T22:03:10.0447847Z OK
2023-01-11T22:03:10.0447852Z 
2023-01-11T22:03:10.0447936Z Generating XML reports...
2023-01-11T22:03:10.0448308Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213003.xml
2023-01-11T22:03:10.0448583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0448708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0448989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0449145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0449333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsez8g332
2023-01-11T22:03:10.0449575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsez8g332/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0449827Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0449833Z 
2023-01-11T22:03:10.0449906Z Running tests...
2023-01-11T22:03:10.0450102Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0450355Z   test_backward_node_failure_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8270
2023-01-11T22:03:10.0450507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8271
2023-01-11T22:03:10.0450659Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8272
2023-01-11T22:03:10.0450802Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8273
2023-01-11T22:03:10.0451075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0451195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0451476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0451608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0451794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7yzs3mn
2023-01-11T22:03:10.0451985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7yzs3mn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0452258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0452371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0452654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0452788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0452970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7a_wk51h
2023-01-11T22:03:10.0453165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7a_wk51h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0453437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0453562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0453844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0453981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0454151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1s8vl0vr
2023-01-11T22:03:10.0454347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1s8vl0vr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0454619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0454741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0455021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0455155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0455336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfb_4ayod
2023-01-11T22:03:10.0455528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfb_4ayod/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0455726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0455908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0456069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0456229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0456405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0456582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0456754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0456925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0457233Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0457524Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0457820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0458111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0458379Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0458643Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0458914Z [W tensorpipe_agent.cpp:939] RPC agent for worker0 encountered error when reading incoming response from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0459167Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0459428Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0459644Z [E container.cpp:256] Could not release Dist Autograd Context on node 3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0459928Z [W tensorpipe_agent.cpp:624] RPC agent for worker2 won't send response to request #6 to worker0, as the agent is shutting down
2023-01-11T22:03:10.0460228Z [W tensorpipe_agent.cpp:915] RPC agent for worker3 encountered error when sending outgoing request #4 to worker2: ECONNREFUSED: connection refused (this error originated at tensorpipe/transport/uv/connection_impl.cc:62)
2023-01-11T22:03:10.0460489Z [W tensorpipe_agent.cpp:915] RPC agent for worker1 encountered error when sending outgoing request #5 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0460750Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #10 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0460998Z [W tensorpipe_agent.cpp:915] RPC agent for worker1 encountered error when sending outgoing request #7 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0461260Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0461583Z [W tensorpipe_agent.cpp:678] RPC agent for worker1 encountered error when sending response to request #11 to worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0461952Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0462208Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0462471Z [W tensorpipe_agent.cpp:939] RPC agent for worker1 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0462732Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0462952Z [E container.cpp:256] Could not release Dist Autograd Context on node 1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0463213Z [W tensorpipe_agent.cpp:915] RPC agent for worker1 encountered error when sending outgoing request #8 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0463470Z [W tensorpipe_agent.cpp:915] RPC agent for worker3 encountered error when sending outgoing request #6 to worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0463537Z ok (2.747s)
2023-01-11T22:03:10.0463542Z 
2023-01-11T22:03:10.0463744Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0463809Z Ran 1 test in 2.748s
2023-01-11T22:03:10.0463829Z 
2023-01-11T22:03:10.0463878Z OK
2023-01-11T22:03:10.0463883Z 
2023-01-11T22:03:10.0463968Z Generating XML reports...
2023-01-11T22:03:10.0464341Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213008.xml
2023-01-11T22:03:10.0464620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0464745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0465030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0465170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0465356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpml5dz5nw
2023-01-11T22:03:10.0465543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpml5dz5nw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0465804Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0465810Z 
2023-01-11T22:03:10.0465881Z Running tests...
2023-01-11T22:03:10.0466080Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0466327Z   test_backward_python_udf_error (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8515
2023-01-11T22:03:10.0466484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8516
2023-01-11T22:03:10.0466638Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8517
2023-01-11T22:03:10.0466791Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8518
2023-01-11T22:03:10.0467070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0467185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0467570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0467743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0467930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprx_w2tym
2023-01-11T22:03:10.0468207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0468333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0468618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0468757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0468929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1cflozip
2023-01-11T22:03:10.0469127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprx_w2tym/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0469322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1cflozip/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0469599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0469724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0470003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0470137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0470323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggv24_lk
2023-01-11T22:03:10.0470518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggv24_lk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0470780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0470903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0471187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0471378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0471566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xzjwlk7
2023-01-11T22:03:10.0471758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xzjwlk7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0471921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0472085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0472236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0472400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0472470Z ok (2.642s)
2023-01-11T22:03:10.0472478Z 
2023-01-11T22:03:10.0472677Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0472754Z Ran 1 test in 2.642s
2023-01-11T22:03:10.0472760Z 
2023-01-11T22:03:10.0472821Z OK
2023-01-11T22:03:10.0472826Z 
2023-01-11T22:03:10.0472909Z Generating XML reports...
2023-01-11T22:03:10.0473283Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213012.xml
2023-01-11T22:03:10.0473558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0473671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0473953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0474124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0474341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjwv4y_l
2023-01-11T22:03:10.0474543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjwv4y_l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0474800Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0474805Z 
2023-01-11T22:03:10.0474880Z Running tests...
2023-01-11T22:03:10.0475077Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0475297Z   test_backward_rref (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8758
2023-01-11T22:03:10.0475452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8759
2023-01-11T22:03:10.0475609Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8760
2023-01-11T22:03:10.0475761Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8761
2023-01-11T22:03:10.0476038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0476163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0476446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0476620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0476807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfdkv0kmp
2023-01-11T22:03:10.0476991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfdkv0kmp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0477266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0477391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0477674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0477811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0477994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplrfzt00m
2023-01-11T22:03:10.0478189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplrfzt00m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0478461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0478571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0478927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0479071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0479258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptc7625k1
2023-01-11T22:03:10.0479453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptc7625k1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0479726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0479851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0480132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0480268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0480440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4grq3ylo
2023-01-11T22:03:10.0480635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4grq3ylo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0480839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0481042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0481207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0481369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0481440Z ok (2.650s)
2023-01-11T22:03:10.0481446Z 
2023-01-11T22:03:10.0481645Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0481708Z Ran 1 test in 2.650s
2023-01-11T22:03:10.0481726Z 
2023-01-11T22:03:10.0481775Z OK
2023-01-11T22:03:10.0481779Z 
2023-01-11T22:03:10.0481863Z Generating XML reports...
2023-01-11T22:03:10.0482234Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213017.xml
2023-01-11T22:03:10.0482509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0482638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0482921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0483056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0483243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsleqeu4l
2023-01-11T22:03:10.0483425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsleqeu4l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0483679Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0483685Z 
2023-01-11T22:03:10.0483759Z Running tests...
2023-01-11T22:03:10.0483958Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0484203Z   test_backward_rref_multi (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9001
2023-01-11T22:03:10.0484362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9002
2023-01-11T22:03:10.0484516Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9003
2023-01-11T22:03:10.0484668Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9004
2023-01-11T22:03:10.0484939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0485050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0485333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0485467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0485654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggv92iyt
2023-01-11T22:03:10.0485852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggv92iyt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0486126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0486250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0486530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0486652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0486836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47c2bczp
2023-01-11T22:03:10.0487028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47c2bczp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0487336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0487488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0487772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0487907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0488091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeim4zhq6
2023-01-11T22:03:10.0488285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeim4zhq6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0488544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0488667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0488949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0489084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0489267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbdvp7z7
2023-01-11T22:03:10.0489464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbdvp7z7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0489632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0489795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0489946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0490108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0490175Z ok (2.646s)
2023-01-11T22:03:10.0490183Z 
2023-01-11T22:03:10.0490386Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0490462Z Ran 1 test in 2.646s
2023-01-11T22:03:10.0490467Z 
2023-01-11T22:03:10.0490528Z OK
2023-01-11T22:03:10.0490536Z 
2023-01-11T22:03:10.0490619Z Generating XML reports...
2023-01-11T22:03:10.0490989Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213021.xml
2023-01-11T22:03:10.0491262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0491373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0491657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0491792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0491978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpom2c_xd_
2023-01-11T22:03:10.0492176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpom2c_xd_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0492432Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0492438Z 
2023-01-11T22:03:10.0492514Z Running tests...
2023-01-11T22:03:10.0492712Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0492942Z   test_backward_rref_nested (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9244
2023-01-11T22:03:10.0493098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9245
2023-01-11T22:03:10.0493250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9246
2023-01-11T22:03:10.0493401Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9247
2023-01-11T22:03:10.0493709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0493836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0494163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0494301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0494490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3m2swhd
2023-01-11T22:03:10.0494674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3m2swhd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0494948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0495073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0495354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0495492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0495679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0z4ygivm
2023-01-11T22:03:10.0495875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0z4ygivm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0496146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0496270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0496537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0496673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0496858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8psi6aeq
2023-01-11T22:03:10.0497055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8psi6aeq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0497332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0497455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0497736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0497872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0498043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8k4_shfq
2023-01-11T22:03:10.0498238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8k4_shfq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0498403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0498571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0498734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0498899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0498968Z ok (2.745s)
2023-01-11T22:03:10.0498973Z 
2023-01-11T22:03:10.0499173Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0499235Z Ran 1 test in 2.745s
2023-01-11T22:03:10.0499253Z 
2023-01-11T22:03:10.0499300Z OK
2023-01-11T22:03:10.0499305Z 
2023-01-11T22:03:10.0499388Z Generating XML reports...
2023-01-11T22:03:10.0499758Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213026.xml
2023-01-11T22:03:10.0500033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0500189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0500498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0500637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0500819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fl9ouzv
2023-01-11T22:03:10.0501002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fl9ouzv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0501259Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0501265Z 
2023-01-11T22:03:10.0501338Z Running tests...
2023-01-11T22:03:10.0501537Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0501907Z   test_backward_simple (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9487
2023-01-11T22:03:10.0502068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9488
2023-01-11T22:03:10.0502225Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9489
2023-01-11T22:03:10.0502380Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9490
2023-01-11T22:03:10.0502659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0502771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0503053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0503189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0503376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfby77w2
2023-01-11T22:03:10.0503574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfby77w2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0503851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0503979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0504260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0504382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0504569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfltjd86
2023-01-11T22:03:10.0504765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfltjd86/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0505038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0505163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0505445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0505581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0505766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqz_s8njb
2023-01-11T22:03:10.0505960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqz_s8njb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0506219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0506342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0506621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0506756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0507002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvzw5sbma
2023-01-11T22:03:10.0507234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvzw5sbma/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0507402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0507568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0507722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0507884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0507952Z ok (2.647s)
2023-01-11T22:03:10.0507958Z 
2023-01-11T22:03:10.0508162Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0508239Z Ran 1 test in 2.647s
2023-01-11T22:03:10.0508245Z 
2023-01-11T22:03:10.0508308Z OK
2023-01-11T22:03:10.0508315Z 
2023-01-11T22:03:10.0508399Z Generating XML reports...
2023-01-11T22:03:10.0508774Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213031.xml
2023-01-11T22:03:10.0509048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0509159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0509444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0509581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0509768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ozmeub4
2023-01-11T22:03:10.0509965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ozmeub4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0510224Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0510232Z 
2023-01-11T22:03:10.0510305Z Running tests...
2023-01-11T22:03:10.0510506Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0510752Z   test_backward_simple_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9730
2023-01-11T22:03:10.0510895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9731
2023-01-11T22:03:10.0511050Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9732
2023-01-11T22:03:10.0511201Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9733
2023-01-11T22:03:10.0511475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0511600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0511885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0512023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0512209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0ub9mme
2023-01-11T22:03:10.0512393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0ub9mme/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0512668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0512791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0513073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0513208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0513391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwh_5r8n5
2023-01-11T22:03:10.0513621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwh_5r8n5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0513919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0514043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0514308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0514443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0514625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsx1fg_e7
2023-01-11T22:03:10.0514895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0515017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0515303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0515438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0515624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29ef3yp8
2023-01-11T22:03:10.0515802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsx1fg_e7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0515990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29ef3yp8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0516155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0516319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0516482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0516649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0516716Z ok (2.634s)
2023-01-11T22:03:10.0516721Z 
2023-01-11T22:03:10.0516922Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0516997Z Ran 1 test in 2.635s
2023-01-11T22:03:10.0517003Z 
2023-01-11T22:03:10.0517051Z OK
2023-01-11T22:03:10.0517056Z 
2023-01-11T22:03:10.0517139Z Generating XML reports...
2023-01-11T22:03:10.0517514Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213035.xml
2023-01-11T22:03:10.0517787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0517913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0518193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0518330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0518515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48h5u6_f
2023-01-11T22:03:10.0518696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48h5u6_f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0519036Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0519043Z 
2023-01-11T22:03:10.0519122Z Running tests...
2023-01-11T22:03:10.0519321Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0519574Z   test_backward_simple_script_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9973
2023-01-11T22:03:10.0519732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9974
2023-01-11T22:03:10.0519887Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9975
2023-01-11T22:03:10.0520078Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9976
2023-01-11T22:03:10.0520384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0520497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0520781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0520917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0521104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3v74exkt
2023-01-11T22:03:10.0521297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3v74exkt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0521567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0521693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0521977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0522098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0522284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwcd0cyk
2023-01-11T22:03:10.0522479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwcd0cyk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0522746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0522871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0523149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0523286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0523473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshj6jpa3
2023-01-11T22:03:10.0523669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshj6jpa3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0523927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0524050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0524328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0524464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0524649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60r3bgjt
2023-01-11T22:03:10.0524842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60r3bgjt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0525009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0525176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0525338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0525489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0525557Z ok (2.846s)
2023-01-11T22:03:10.0525563Z 
2023-01-11T22:03:10.0525761Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0525837Z Ran 1 test in 2.846s
2023-01-11T22:03:10.0525842Z 
2023-01-11T22:03:10.0525903Z OK
2023-01-11T22:03:10.0525908Z 
2023-01-11T22:03:10.0525992Z Generating XML reports...
2023-01-11T22:03:10.0526365Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213040.xml
2023-01-11T22:03:10.0526677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0526817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0527101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0527235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0527418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58bwykx2
2023-01-11T22:03:10.0527614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58bwykx2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0527872Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0527878Z 
2023-01-11T22:03:10.0527951Z Running tests...
2023-01-11T22:03:10.0528148Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0528394Z   test_backward_simple_self (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10216
2023-01-11T22:03:10.0528542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10217
2023-01-11T22:03:10.0528699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10218
2023-01-11T22:03:10.0528853Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10219
2023-01-11T22:03:10.0529127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0529251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0529534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0529668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0529859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8g8ddg36
2023-01-11T22:03:10.0530044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8g8ddg36/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0530317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0530440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0530720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0530855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0531038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpant1njir
2023-01-11T22:03:10.0531232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpant1njir/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0531506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0531629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0531900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0532033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0532217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3yyaeaa
2023-01-11T22:03:10.0532410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3yyaeaa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0532681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0532804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0533083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0533250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0533449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8l1xsm0
2023-01-11T22:03:10.0533644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8l1xsm0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0533808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0533973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0534136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0534299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0534366Z ok (2.641s)
2023-01-11T22:03:10.0534372Z 
2023-01-11T22:03:10.0534572Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0534653Z Ran 1 test in 2.641s
2023-01-11T22:03:10.0534658Z 
2023-01-11T22:03:10.0534707Z OK
2023-01-11T22:03:10.0534712Z 
2023-01-11T22:03:10.0534799Z Generating XML reports...
2023-01-11T22:03:10.0535166Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213045.xml
2023-01-11T22:03:10.0535442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0535565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0535844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0535979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0536163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrh1o9r2
2023-01-11T22:03:10.0536347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrh1o9r2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0536604Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0536610Z 
2023-01-11T22:03:10.0536683Z Running tests...
2023-01-11T22:03:10.0536879Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0537129Z   test_backward_unused_send_function (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10459
2023-01-11T22:03:10.0537287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10460
2023-01-11T22:03:10.0537442Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10461
2023-01-11T22:03:10.0537595Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10462
2023-01-11T22:03:10.0537868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0537983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0538269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0538404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0538590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzq2baa8k
2023-01-11T22:03:10.0538786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzq2baa8k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0539063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0539187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0539468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0539658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0539875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwbkld1rb
2023-01-11T22:03:10.0540074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwbkld1rb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0540348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0540471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0540753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0540888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0541073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplz6ax_7x
2023-01-11T22:03:10.0541271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplz6ax_7x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0541533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0541775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0542058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0542194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0542379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhsw0n65
2023-01-11T22:03:10.0542576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhsw0n65/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0542743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0542908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0543074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0543226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0543296Z ok (12.675s)
2023-01-11T22:03:10.0543303Z 
2023-01-11T22:03:10.0543503Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0543581Z Ran 1 test in 12.675s
2023-01-11T22:03:10.0543587Z 
2023-01-11T22:03:10.0543648Z OK
2023-01-11T22:03:10.0543653Z 
2023-01-11T22:03:10.0543738Z Generating XML reports...
2023-01-11T22:03:10.0544114Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213049.xml
2023-01-11T22:03:10.0544390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0544502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0544787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0544925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0545111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxuzdy_ic
2023-01-11T22:03:10.0545306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxuzdy_ic/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0545563Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0545569Z 
2023-01-11T22:03:10.0545643Z Running tests...
2023-01-11T22:03:10.0545840Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0546605Z   test_backward_unused_tensors (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77556 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.320s)
2023-01-11T22:03:10.0546649Z 
2023-01-11T22:03:10.0546850Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0546913Z Ran 1 test in 0.320s
2023-01-11T22:03:10.0546918Z 
2023-01-11T22:03:10.0546992Z OK (skipped=1)
2023-01-11T22:03:10.0546997Z 
2023-01-11T22:03:10.0547082Z Generating XML reports...
2023-01-11T22:03:10.0547458Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213104.xml
2023-01-11T22:03:10.0547734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0547862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0548149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0548289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0548480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0pf1uye
2023-01-11T22:03:10.0548664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0pf1uye/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0548920Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0548925Z 
2023-01-11T22:03:10.0549001Z Running tests...
2023-01-11T22:03:10.0549201Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0549448Z   test_backward_verify_hooks (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10716
2023-01-11T22:03:10.0549608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10717
2023-01-11T22:03:10.0549768Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10718
2023-01-11T22:03:10.0549924Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10719
2023-01-11T22:03:10.0550188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0550312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0550594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0550728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0550914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpox9nesq8
2023-01-11T22:03:10.0551112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpox9nesq8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0551387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0551514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0551798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0551919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0552102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_oorha1
2023-01-11T22:03:10.0552298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_oorha1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0552573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0552695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0552975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0553141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0553355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbw0gcm19
2023-01-11T22:03:10.0553552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbw0gcm19/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0553814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0553939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0554218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0554353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0554537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvse8y2c_
2023-01-11T22:03:10.0554733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvse8y2c_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0554900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0555065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0555214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0555376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0555443Z ok (2.740s)
2023-01-11T22:03:10.0555449Z 
2023-01-11T22:03:10.0555645Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0555720Z Ran 1 test in 2.740s
2023-01-11T22:03:10.0555725Z 
2023-01-11T22:03:10.0555785Z OK
2023-01-11T22:03:10.0555791Z 
2023-01-11T22:03:10.0555874Z Generating XML reports...
2023-01-11T22:03:10.0556244Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213106.xml
2023-01-11T22:03:10.0556522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0556635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0556920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0557056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0557241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2un8htez
2023-01-11T22:03:10.0557439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2un8htez/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0557691Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0557696Z 
2023-01-11T22:03:10.0557772Z Running tests...
2023-01-11T22:03:10.0557972Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0558206Z   test_backward_without_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10959
2023-01-11T22:03:10.0558364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10960
2023-01-11T22:03:10.0558520Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10961
2023-01-11T22:03:10.0558674Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10962
2023-01-11T22:03:10.0559027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0559154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0559440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0559614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0559831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9g4tx0lb
2023-01-11T22:03:10.0560015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9g4tx0lb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0560292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0560420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0560702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0560838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0561022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6s2tomq3
2023-01-11T22:03:10.0561217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6s2tomq3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0561494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0561604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0561884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0562021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0562204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp32b1wnza
2023-01-11T22:03:10.0562398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp32b1wnza/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0562670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0562792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0563073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0563211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0563382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47esi88j
2023-01-11T22:03:10.0563579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47esi88j/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0563742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0563909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0564069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0564231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0564301Z ok (2.650s)
2023-01-11T22:03:10.0564306Z 
2023-01-11T22:03:10.0564502Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0564565Z Ran 1 test in 2.650s
2023-01-11T22:03:10.0564584Z 
2023-01-11T22:03:10.0564632Z OK
2023-01-11T22:03:10.0564637Z 
2023-01-11T22:03:10.0564720Z Generating XML reports...
2023-01-11T22:03:10.0565088Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213111.xml
2023-01-11T22:03:10.0565362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0565488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0565771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0565907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0566124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppioeiqh5
2023-01-11T22:03:10.0566371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppioeiqh5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0566632Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0566638Z 
2023-01-11T22:03:10.0566712Z Running tests...
2023-01-11T22:03:10.0566908Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0567151Z   test_backward_without_rpc (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11186
2023-01-11T22:03:10.0567309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11187
2023-01-11T22:03:10.0567464Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11188
2023-01-11T22:03:10.0567622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11189
2023-01-11T22:03:10.0567901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0568016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0568301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0568437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0568624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpovj8bnug
2023-01-11T22:03:10.0568823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpovj8bnug/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0569096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0569221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0569508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0569632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0569817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeit24mzq
2023-01-11T22:03:10.0570092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0570217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0570495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0570630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0570813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzl9an6yu
2023-01-11T22:03:10.0571007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeit24mzq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0571201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzl9an6yu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0571463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0571586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0571865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0572002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0572183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_n2tutpd
2023-01-11T22:03:10.0572378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_n2tutpd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0572545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0572742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0572921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0573086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0573155Z ok (2.646s)
2023-01-11T22:03:10.0573160Z 
2023-01-11T22:03:10.0573359Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0573435Z Ran 1 test in 2.646s
2023-01-11T22:03:10.0573440Z 
2023-01-11T22:03:10.0573501Z OK
2023-01-11T22:03:10.0573507Z 
2023-01-11T22:03:10.0573590Z Generating XML reports...
2023-01-11T22:03:10.0573964Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213115.xml
2023-01-11T22:03:10.0574237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0574351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0574636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0574772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0574959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3mt9pzm
2023-01-11T22:03:10.0575157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3mt9pzm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0575416Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0575422Z 
2023-01-11T22:03:10.0575494Z Running tests...
2023-01-11T22:03:10.0575692Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0575931Z   test_backwards_nested_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11429
2023-01-11T22:03:10.0576091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11430
2023-01-11T22:03:10.0576249Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11431
2023-01-11T22:03:10.0576403Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11432
2023-01-11T22:03:10.0576678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0576803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0577085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0577222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0577406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1ip0lfx
2023-01-11T22:03:10.0577590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1ip0lfx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0577863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0577986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0578271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0578405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0578590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnr8av9z
2023-01-11T22:03:10.0578784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnr8av9z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0579057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0579204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0579517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0579653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0579836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpagaoeb6h
2023-01-11T22:03:10.0580031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpagaoeb6h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0580304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0580429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0580709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0580835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0581024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplh9nbrcd
2023-01-11T22:03:10.0581222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplh9nbrcd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0581387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0581552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0581834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0581999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0582070Z ok (2.746s)
2023-01-11T22:03:10.0582076Z 
2023-01-11T22:03:10.0582262Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0582342Z Ran 1 test in 2.746s
2023-01-11T22:03:10.0582350Z 
2023-01-11T22:03:10.0582412Z OK
2023-01-11T22:03:10.0582417Z 
2023-01-11T22:03:10.0582503Z Generating XML reports...
2023-01-11T22:03:10.0582876Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213120.xml
2023-01-11T22:03:10.0583154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0583280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0583566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0583705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0583880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgb5lad61
2023-01-11T22:03:10.0584079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgb5lad61/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0584337Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0584342Z 
2023-01-11T22:03:10.0584420Z Running tests...
2023-01-11T22:03:10.0584620Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0584778Z   test_clean_context_during_backward (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:10.0585125Z This test simulates the situation where the 'backward' call might throw ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11672
2023-01-11T22:03:10.0585285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11673
2023-01-11T22:03:10.0585429Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11674
2023-01-11T22:03:10.0585583Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11675
2023-01-11T22:03:10.0585860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0586040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0586375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0586516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0586705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpksbjr9_u
2023-01-11T22:03:10.0586906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpksbjr9_u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0587180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0587292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0587578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0587718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0587907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4kmx264d
2023-01-11T22:03:10.0588105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4kmx264d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0588379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0588505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0588788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0588925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0589096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_gbnsopi
2023-01-11T22:03:10.0589295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_gbnsopi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0589573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0589698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0589980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0590117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0590303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4lr6t14
2023-01-11T22:03:10.0590499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4lr6t14/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0590652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0590816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0590983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0591150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0591331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0591634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0591812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0591986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0592161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0592448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0592816Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0593112Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0593384Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0593652Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0593919Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0594185Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0594443Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0594703Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0594961Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0595216Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0595473Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0595726Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0595969Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0596223Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0596296Z ok (2.952s)
2023-01-11T22:03:10.0596301Z 
2023-01-11T22:03:10.0596504Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0596581Z Ran 1 test in 2.952s
2023-01-11T22:03:10.0596587Z 
2023-01-11T22:03:10.0596649Z OK
2023-01-11T22:03:10.0596654Z 
2023-01-11T22:03:10.0596742Z Generating XML reports...
2023-01-11T22:03:10.0597114Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213125.xml
2023-01-11T22:03:10.0597390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0597503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0597791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0597927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0598112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3bblm2a
2023-01-11T22:03:10.0598345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3bblm2a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0598628Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0598635Z 
2023-01-11T22:03:10.0598712Z Running tests...
2023-01-11T22:03:10.0598982Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0599235Z   test_context_cleanup_nested_rpc (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11911
2023-01-11T22:03:10.0599381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11912
2023-01-11T22:03:10.0599538Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11913
2023-01-11T22:03:10.0599693Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11914
2023-01-11T22:03:10.0599974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0600101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0600394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0600530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0600719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsq5611ew
2023-01-11T22:03:10.0600900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsq5611ew/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0601174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0601297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0601578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0601717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0601906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsm89ewk5
2023-01-11T22:03:10.0602100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsm89ewk5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0602370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0602493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0602761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0602893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0603078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_10770c
2023-01-11T22:03:10.0603269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_10770c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0603546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0603668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0603949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0604082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0604252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79__t2x4
2023-01-11T22:03:10.0604441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79__t2x4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0604605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0604768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0604969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0605166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0605344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0605519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0605693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0605984Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0606280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0606577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0606753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0607053Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0607121Z ok (2.847s)
2023-01-11T22:03:10.0607127Z 
2023-01-11T22:03:10.0607326Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0607402Z Ran 1 test in 2.847s
2023-01-11T22:03:10.0607408Z 
2023-01-11T22:03:10.0607469Z OK
2023-01-11T22:03:10.0607475Z 
2023-01-11T22:03:10.0607546Z Generating XML reports...
2023-01-11T22:03:10.0607919Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213129.xml
2023-01-11T22:03:10.0608200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0608330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0608614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0608749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0608935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2bi04okf
2023-01-11T22:03:10.0609129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2bi04okf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0609384Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0609389Z 
2023-01-11T22:03:10.0609449Z Running tests...
2023-01-11T22:03:10.0609645Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0609889Z   test_context_cleanup_no_tensors (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12146
2023-01-11T22:03:10.0610049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12147
2023-01-11T22:03:10.0610203Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12148
2023-01-11T22:03:10.0610355Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12149
2023-01-11T22:03:10.0610631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0610753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0611019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0611151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0611337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuibxtrhw
2023-01-11T22:03:10.0611568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuibxtrhw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0611870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0611994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0612276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0612412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0612593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp41_i31u3
2023-01-11T22:03:10.0612771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp41_i31u3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0613039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0613163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0613443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0613577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0613760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8v86x2pr
2023-01-11T22:03:10.0613953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8v86x2pr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0614223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0614333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0614612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0614747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0614929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaxci63qc
2023-01-11T22:03:10.0615125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaxci63qc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0615288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0615454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0615614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0615774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0615938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0616113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0616288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0616587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0616887Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0617061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0617360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0617656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0617724Z ok (2.845s)
2023-01-11T22:03:10.0617730Z 
2023-01-11T22:03:10.0617946Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0618022Z Ran 1 test in 2.845s
2023-01-11T22:03:10.0618027Z 
2023-01-11T22:03:10.0618086Z OK
2023-01-11T22:03:10.0618091Z 
2023-01-11T22:03:10.0618201Z Generating XML reports...
2023-01-11T22:03:10.0618576Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213134.xml
2023-01-11T22:03:10.0618851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0618974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0619256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0619389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0619565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmprpxld4
2023-01-11T22:03:10.0619762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmprpxld4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0620017Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0620023Z 
2023-01-11T22:03:10.0620095Z Running tests...
2023-01-11T22:03:10.0620292Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0620545Z   test_context_cleanup_tensor_no_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12381
2023-01-11T22:03:10.0620701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12382
2023-01-11T22:03:10.0620856Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12383
2023-01-11T22:03:10.0620997Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12384
2023-01-11T22:03:10.0621274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0621400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0621802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0621940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0622127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcc9lxkxo
2023-01-11T22:03:10.0622327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcc9lxkxo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0622605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0622730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0622999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0623137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0623326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1kjqoimp
2023-01-11T22:03:10.0623525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1kjqoimp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0623799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0623923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0624205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0624341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0624512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8dbpesz7
2023-01-11T22:03:10.0624777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8dbpesz7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0625089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0625214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0625498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0625634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0625822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpou6foiot
2023-01-11T22:03:10.0626020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpou6foiot/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0626187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0626340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0626509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0626673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0626854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0627030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0627204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0627377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0627680Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0627980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0628268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0628557Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0628623Z ok (2.938s)
2023-01-11T22:03:10.0628629Z 
2023-01-11T22:03:10.0628825Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0628900Z Ran 1 test in 2.938s
2023-01-11T22:03:10.0628906Z 
2023-01-11T22:03:10.0628965Z OK
2023-01-11T22:03:10.0628970Z 
2023-01-11T22:03:10.0629056Z Generating XML reports...
2023-01-11T22:03:10.0629426Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213139.xml
2023-01-11T22:03:10.0629700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0629813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0630098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0630234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0630421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpev1eb17v
2023-01-11T22:03:10.0630619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpev1eb17v/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0630876Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0630882Z 
2023-01-11T22:03:10.0630955Z Running tests...
2023-01-11T22:03:10.0631151Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0631394Z   test_context_cleanup_tensor_with_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12616
2023-01-11T22:03:10.0631625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12617
2023-01-11T22:03:10.0631782Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12618
2023-01-11T22:03:10.0631936Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12619
2023-01-11T22:03:10.0632212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0632338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0632619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0632754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0632940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjadnpj6
2023-01-11T22:03:10.0633203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0633332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0633612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0633747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0633934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxc84x4yy
2023-01-11T22:03:10.0634131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjadnpj6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0634323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxc84x4yy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0634596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0634722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0634991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0635126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0635310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqnf575f
2023-01-11T22:03:10.0635505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqnf575f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0635778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0635905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0636185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0636321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0636492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprx0ttuj0
2023-01-11T22:03:10.0636689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprx0ttuj0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0636853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0637015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0637177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0637338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0637514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0637689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0637896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0638217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0638392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0638688Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0639074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0639372Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0639441Z ok (2.841s)
2023-01-11T22:03:10.0639446Z 
2023-01-11T22:03:10.0639651Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0639729Z Ran 1 test in 2.841s
2023-01-11T22:03:10.0639734Z 
2023-01-11T22:03:10.0639798Z OK
2023-01-11T22:03:10.0639803Z 
2023-01-11T22:03:10.0639877Z Generating XML reports...
2023-01-11T22:03:10.0640251Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213144.xml
2023-01-11T22:03:10.0640531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0640656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0640940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0641078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0641266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ga2pa3a
2023-01-11T22:03:10.0641467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ga2pa3a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0641713Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0641731Z 
2023-01-11T22:03:10.0641792Z Running tests...
2023-01-11T22:03:10.0641991Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0642224Z   test_debug_info (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12851
2023-01-11T22:03:10.0642384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12852
2023-01-11T22:03:10.0642541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12853
2023-01-11T22:03:10.0642696Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12854
2023-01-11T22:03:10.0642971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0643098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0643369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0643505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0643691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplm_671v2
2023-01-11T22:03:10.0643889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplm_671v2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0644162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0644288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0644569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0644750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0644953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqioj7y2e
2023-01-11T22:03:10.0645146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqioj7y2e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0645419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0645544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0645826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0645961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0646146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwbta9qt
2023-01-11T22:03:10.0646346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwbta9qt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0646621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0646732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0647014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0647150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0647335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfh3zk__
2023-01-11T22:03:10.0647530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfh3zk__/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0647693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0647856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0648019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0648185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0648350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0648529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0648832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0649005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0649175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0649466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0649767Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0650057Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0650127Z ok (2.847s)
2023-01-11T22:03:10.0650133Z 
2023-01-11T22:03:10.0650318Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0650393Z Ran 1 test in 2.847s
2023-01-11T22:03:10.0650399Z 
2023-01-11T22:03:10.0650462Z OK
2023-01-11T22:03:10.0650467Z 
2023-01-11T22:03:10.0650552Z Generating XML reports...
2023-01-11T22:03:10.0650925Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213149.xml
2023-01-11T22:03:10.0651198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0651353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0651665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0651789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0651976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptiikerhx
2023-01-11T22:03:10.0652171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptiikerhx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0652426Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0652432Z 
2023-01-11T22:03:10.0652507Z Running tests...
2023-01-11T22:03:10.0652701Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0653413Z   test_dist_autograd_profiling (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77318 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.330s)
2023-01-11T22:03:10.0653421Z 
2023-01-11T22:03:10.0653615Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0653691Z Ran 1 test in 0.330s
2023-01-11T22:03:10.0653696Z 
2023-01-11T22:03:10.0653768Z OK (skipped=1)
2023-01-11T22:03:10.0653773Z 
2023-01-11T22:03:10.0653844Z Generating XML reports...
2023-01-11T22:03:10.0654215Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213153.xml
2023-01-11T22:03:10.0654488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0654617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0654900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0655038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0655227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3gr_f37k
2023-01-11T22:03:10.0655421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3gr_f37k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0655676Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0655681Z 
2023-01-11T22:03:10.0655741Z Running tests...
2023-01-11T22:03:10.0655938Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0656173Z   test_error_in_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13116
2023-01-11T22:03:10.0656331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13117
2023-01-11T22:03:10.0656488Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13118
2023-01-11T22:03:10.0656644Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13119
2023-01-11T22:03:10.0656922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0657046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0657331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0657456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0657639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7yrigdf
2023-01-11T22:03:10.0657835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7yrigdf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0658141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0658296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0658577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0658712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0658900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjpndfi2
2023-01-11T22:03:10.0659084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjpndfi2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0659358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0659483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0659765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0659904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0660091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqyd3m44s
2023-01-11T22:03:10.0660284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqyd3m44s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0660555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0660677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0660944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0661086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0661271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kb_16qk
2023-01-11T22:03:10.0661463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kb_16qk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0661626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0661906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0662073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0662239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0662294Z ok (2.747s)
2023-01-11T22:03:10.0662301Z 
2023-01-11T22:03:10.0662507Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0662584Z Ran 1 test in 2.747s
2023-01-11T22:03:10.0662590Z 
2023-01-11T22:03:10.0662651Z OK
2023-01-11T22:03:10.0662658Z 
2023-01-11T22:03:10.0662745Z Generating XML reports...
2023-01-11T22:03:10.0663120Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213156.xml
2023-01-11T22:03:10.0663395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0663520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0663800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0663923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0664107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97w2_hui
2023-01-11T22:03:10.0664301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97w2_hui/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0664558Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0664616Z 
2023-01-11T22:03:10.0664692Z Running tests...
2023-01-11T22:03:10.0664926Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0665186Z   test_grad_copy_sparse_indices_extra_ref (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13339
2023-01-11T22:03:10.0665397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13340
2023-01-11T22:03:10.0665538Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13341
2023-01-11T22:03:10.0665695Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13342
2023-01-11T22:03:10.0665973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0666099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0666388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0666527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0666717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9yxyf72
2023-01-11T22:03:10.0666913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9yxyf72/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0667186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0667296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0667575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0667711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0667902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzafvtsm9
2023-01-11T22:03:10.0668100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzafvtsm9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0668374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0668495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0668773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0668906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0669078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeajw2klp
2023-01-11T22:03:10.0669271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeajw2klp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0669542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0669666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0669948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0670081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0670264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgt6ml4n
2023-01-11T22:03:10.0670458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgt6ml4n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0670610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0670774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0670938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0671166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0671692Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2023-01-11T22:03:10.0671833Z   "Argument order of nn.functional.embedding_bag was changed. "
2023-01-11T22:03:10.0672325Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2023-01-11T22:03:10.0672461Z   "Argument order of nn.functional.embedding_bag was changed. "
2023-01-11T22:03:10.0672947Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2023-01-11T22:03:10.0673085Z   "Argument order of nn.functional.embedding_bag was changed. "
2023-01-11T22:03:10.0673579Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2023-01-11T22:03:10.0673702Z   "Argument order of nn.functional.embedding_bag was changed. "
2023-01-11T22:03:10.0673771Z ok (2.644s)
2023-01-11T22:03:10.0673778Z 
2023-01-11T22:03:10.0673976Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0674052Z Ran 1 test in 2.645s
2023-01-11T22:03:10.0674057Z 
2023-01-11T22:03:10.0674118Z OK
2023-01-11T22:03:10.0674124Z 
2023-01-11T22:03:10.0674210Z Generating XML reports...
2023-01-11T22:03:10.0674580Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213200.xml
2023-01-11T22:03:10.0674857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0674970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0675253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0675388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0675576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9c_37aox
2023-01-11T22:03:10.0675772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9c_37aox/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0676030Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0676035Z 
2023-01-11T22:03:10.0676110Z Running tests...
2023-01-11T22:03:10.0676309Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0676554Z   test_grad_only_on_return_value (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13594
2023-01-11T22:03:10.0676700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13595
2023-01-11T22:03:10.0676853Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13596
2023-01-11T22:03:10.0677008Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13597
2023-01-11T22:03:10.0677282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0677406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0677689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0677854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0678040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_rb9blq
2023-01-11T22:03:10.0678262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_rb9blq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0678537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0678660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0679020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0679160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0679346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgzuwyfh
2023-01-11T22:03:10.0679545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgzuwyfh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0679825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0679954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0680223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0680359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0680545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10w5y2os
2023-01-11T22:03:10.0680738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10w5y2os/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0681011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0681137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0681421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0681559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0681745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1ci4l5h
2023-01-11T22:03:10.0681925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1ci4l5h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0682090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0682257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0682420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0682582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0682758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0682937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0683114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0683273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0683573Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0683872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0684166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0684458Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0684563Z ok (2.839s)
2023-01-11T22:03:10.0684569Z 
2023-01-11T22:03:10.0684797Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0684876Z Ran 1 test in 2.839s
2023-01-11T22:03:10.0684882Z 
2023-01-11T22:03:10.0684943Z OK
2023-01-11T22:03:10.0684948Z 
2023-01-11T22:03:10.0685019Z Generating XML reports...
2023-01-11T22:03:10.0685391Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213205.xml
2023-01-11T22:03:10.0685666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0685791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0686070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0686206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0686394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmleq65u
2023-01-11T22:03:10.0686591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmleq65u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0686846Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0686852Z 
2023-01-11T22:03:10.0686912Z Running tests...
2023-01-11T22:03:10.0687110Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0687358Z   test_grad_only_on_return_value_remote (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13849
2023-01-11T22:03:10.0687515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13850
2023-01-11T22:03:10.0687668Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13851
2023-01-11T22:03:10.0687828Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13852
2023-01-11T22:03:10.0688105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0688228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0688511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0688633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0688819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwk7bqq6
2023-01-11T22:03:10.0689014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwk7bqq6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0689288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0689414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0689696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0689834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0690021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7tiyitk
2023-01-11T22:03:10.0690204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7tiyitk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0690472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0690595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0690874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0691009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0691227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzh6y4byz
2023-01-11T22:03:10.0691450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzh6y4byz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0691725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0691848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0692119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0692252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0692435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_w_2s3d
2023-01-11T22:03:10.0692625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_w_2s3d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0692791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0692958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0693120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0693280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0693446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0693620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0693793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0694093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0694267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0694569Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0694866Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0695157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0695228Z ok (2.839s)
2023-01-11T22:03:10.0695234Z 
2023-01-11T22:03:10.0695421Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0695499Z Ran 1 test in 2.839s
2023-01-11T22:03:10.0695504Z 
2023-01-11T22:03:10.0695565Z OK
2023-01-11T22:03:10.0695571Z 
2023-01-11T22:03:10.0695659Z Generating XML reports...
2023-01-11T22:03:10.0696026Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213210.xml
2023-01-11T22:03:10.0696308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0696435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0696720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0696856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0697031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5snyi3i
2023-01-11T22:03:10.0697228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5snyi3i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0697485Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0697491Z 
2023-01-11T22:03:10.0697566Z Running tests...
2023-01-11T22:03:10.0697796Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0698075Z   test_graph_for_builtin_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14104
2023-01-11T22:03:10.0698235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14105
2023-01-11T22:03:10.0698388Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14106
2023-01-11T22:03:10.0698542Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14107
2023-01-11T22:03:10.0698808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0698934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0699221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0699362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0699552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnisqzgm2
2023-01-11T22:03:10.0699753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnisqzgm2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0700029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0700156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0700424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0700561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0700751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsr1c5vrf
2023-01-11T22:03:10.0700947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsr1c5vrf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0701223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0701351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0701635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0701888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0702074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3r0b_q06
2023-01-11T22:03:10.0702256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3r0b_q06/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0702532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0702657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0702939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0703077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0703263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpti39po7m
2023-01-11T22:03:10.0703459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpti39po7m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0703627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0703780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0703945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0704107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0704284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0704512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0704723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0704901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0705201Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0705503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0705786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0706080Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0706152Z ok (2.837s)
2023-01-11T22:03:10.0706158Z 
2023-01-11T22:03:10.0706359Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0706436Z Ran 1 test in 2.837s
2023-01-11T22:03:10.0706441Z 
2023-01-11T22:03:10.0706502Z OK
2023-01-11T22:03:10.0706507Z 
2023-01-11T22:03:10.0706592Z Generating XML reports...
2023-01-11T22:03:10.0706965Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213214.xml
2023-01-11T22:03:10.0707241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0707352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0707636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0707777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0707967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuel0jfh0
2023-01-11T22:03:10.0708167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuel0jfh0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0708422Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0708428Z 
2023-01-11T22:03:10.0708502Z Running tests...
2023-01-11T22:03:10.0708700Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0708951Z   test_graph_for_builtin_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14339
2023-01-11T22:03:10.0709096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14340
2023-01-11T22:03:10.0709254Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14341
2023-01-11T22:03:10.0709411Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14342
2023-01-11T22:03:10.0709689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0709812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0710098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0710236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0710421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx69xuox3
2023-01-11T22:03:10.0710603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx69xuox3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0710875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0711031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0711313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0711476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0711660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk5b1vje0
2023-01-11T22:03:10.0711856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk5b1vje0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0712129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0712250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0712516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0712649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0712834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50d5do_1
2023-01-11T22:03:10.0713031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50d5do_1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0713304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0713428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0713706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0713838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0714010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0sazgac3
2023-01-11T22:03:10.0714201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0sazgac3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0714367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0714535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0714696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0714856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0715034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0715211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0715383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0715671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0715969Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0716144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0716442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0716736Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0716803Z ok (2.941s)
2023-01-11T22:03:10.0716809Z 
2023-01-11T22:03:10.0717005Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0717080Z Ran 1 test in 2.941s
2023-01-11T22:03:10.0717085Z 
2023-01-11T22:03:10.0717145Z OK
2023-01-11T22:03:10.0717150Z 
2023-01-11T22:03:10.0717222Z Generating XML reports...
2023-01-11T22:03:10.0717590Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213219.xml
2023-01-11T22:03:10.0717928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0718054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0718333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0718469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0718659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxa6bv41
2023-01-11T22:03:10.0718921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxa6bv41/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0719181Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0719187Z 
2023-01-11T22:03:10.0719248Z Running tests...
2023-01-11T22:03:10.0719449Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0719698Z   test_graph_for_py_nested_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14574
2023-01-11T22:03:10.0719857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14575
2023-01-11T22:03:10.0720014Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14576
2023-01-11T22:03:10.0720169Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14577
2023-01-11T22:03:10.0720444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0720567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0720836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0720977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0721161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2i10uw4c
2023-01-11T22:03:10.0721356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2i10uw4c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0721629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0721752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0722032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0722168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0722350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3xjr9l1
2023-01-11T22:03:10.0722529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3xjr9l1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0722803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0722926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0723208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0723342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0723525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8go5bpko
2023-01-11T22:03:10.0723720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8go5bpko/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0723991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0724102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0724418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0724594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0724779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgi6m69u6
2023-01-11T22:03:10.0724971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgi6m69u6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0725133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0725296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0725458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0725619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0725782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0725961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0726264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0726437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0726608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0726902Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0727198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0727486Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0727556Z ok (2.951s)
2023-01-11T22:03:10.0727563Z 
2023-01-11T22:03:10.0727749Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0727826Z Ran 1 test in 2.951s
2023-01-11T22:03:10.0727832Z 
2023-01-11T22:03:10.0727893Z OK
2023-01-11T22:03:10.0727898Z 
2023-01-11T22:03:10.0727983Z Generating XML reports...
2023-01-11T22:03:10.0728350Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213224.xml
2023-01-11T22:03:10.0728625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0728749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0729028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0729161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0729336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyq5cet84
2023-01-11T22:03:10.0729535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyq5cet84/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0729789Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0729794Z 
2023-01-11T22:03:10.0729870Z Running tests...
2023-01-11T22:03:10.0730068Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0730319Z   test_graph_for_py_nested_call_itself (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14809
2023-01-11T22:03:10.0730475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14810
2023-01-11T22:03:10.0730630Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14811
2023-01-11T22:03:10.0730808Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14812
2023-01-11T22:03:10.0731117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0731242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0731526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0731661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0731847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg52cl6uk
2023-01-11T22:03:10.0732044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg52cl6uk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0732317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0732443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0732712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0732848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0733033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rx14vpa
2023-01-11T22:03:10.0733227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rx14vpa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0733498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0733620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0733899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0734036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0734211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgzy1sqy
2023-01-11T22:03:10.0734406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgzy1sqy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0734680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0734804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0735082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0735216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0735399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09q7ua37
2023-01-11T22:03:10.0735590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09q7ua37/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0735757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0735911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0736072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0736232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0736412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0736591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0736765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0737064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0737270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0737599Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0737883Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0738173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0738239Z ok (2.947s)
2023-01-11T22:03:10.0738245Z 
2023-01-11T22:03:10.0738442Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0738517Z Ran 1 test in 2.947s
2023-01-11T22:03:10.0738523Z 
2023-01-11T22:03:10.0738586Z OK
2023-01-11T22:03:10.0738591Z 
2023-01-11T22:03:10.0738675Z Generating XML reports...
2023-01-11T22:03:10.0739045Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213229.xml
2023-01-11T22:03:10.0739322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0739434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0739715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0739852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0740036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30bzo8_5
2023-01-11T22:03:10.0740231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30bzo8_5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0740491Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0740497Z 
2023-01-11T22:03:10.0740571Z Running tests...
2023-01-11T22:03:10.0740772Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0741015Z   test_graph_for_py_nested_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15044
2023-01-11T22:03:10.0741173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15045
2023-01-11T22:03:10.0741332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15046
2023-01-11T22:03:10.0741484Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15047
2023-01-11T22:03:10.0741881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0742008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0742294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0742432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0742619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcamcpvpr
2023-01-11T22:03:10.0742806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcamcpvpr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0743080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0743204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0743485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0743622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0743807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkz6zwk7
2023-01-11T22:03:10.0744082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0744266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0744616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0744742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0744926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4nagft7x
2023-01-11T22:03:10.0745121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkz6zwk7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0745311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4nagft7x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0745589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0745714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0745996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0746132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0746303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_os6p8f
2023-01-11T22:03:10.0746498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_os6p8f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0746661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0746824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0746985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0747147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0747323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0747504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0747810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0747967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0748139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0748435Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0748733Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0749025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0749096Z ok (2.842s)
2023-01-11T22:03:10.0749102Z 
2023-01-11T22:03:10.0749298Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0749372Z Ran 1 test in 2.843s
2023-01-11T22:03:10.0749378Z 
2023-01-11T22:03:10.0749438Z OK
2023-01-11T22:03:10.0749443Z 
2023-01-11T22:03:10.0749514Z Generating XML reports...
2023-01-11T22:03:10.0749882Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213234.xml
2023-01-11T22:03:10.0750158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0750280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0750563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0750738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0750923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2eubvrt8
2023-01-11T22:03:10.0751148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2eubvrt8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0751393Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0751409Z 
2023-01-11T22:03:10.0751470Z Running tests...
2023-01-11T22:03:10.0751666Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0751925Z   test_graph_for_py_nested_remote_call_itself (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15279
2023-01-11T22:03:10.0752085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15280
2023-01-11T22:03:10.0752243Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15281
2023-01-11T22:03:10.0752401Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15282
2023-01-11T22:03:10.0752682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0752811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0753083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0753222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0753409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkidy3e8
2023-01-11T22:03:10.0753610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkidy3e8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0753886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0754015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0754301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0754439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0754626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9j419e4w
2023-01-11T22:03:10.0754808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9j419e4w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0755082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0755206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0755489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0755627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0755816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4282e_t
2023-01-11T22:03:10.0756014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4282e_t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0756286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0756397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0756680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0756816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0757003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpboppqpb5
2023-01-11T22:03:10.0757202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpboppqpb5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0757402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0757600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0757765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0757925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0758090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0758267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0758567Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0758740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0759003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0759307Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0759606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0759898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0759966Z ok (2.850s)
2023-01-11T22:03:10.0759972Z 
2023-01-11T22:03:10.0760157Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0760234Z Ran 1 test in 2.851s
2023-01-11T22:03:10.0760239Z 
2023-01-11T22:03:10.0760301Z OK
2023-01-11T22:03:10.0760306Z 
2023-01-11T22:03:10.0760392Z Generating XML reports...
2023-01-11T22:03:10.0760765Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213239.xml
2023-01-11T22:03:10.0761047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0761172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0761458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0761596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0761770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwblkd2g
2023-01-11T22:03:10.0761969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwblkd2g/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0762225Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0762230Z 
2023-01-11T22:03:10.0762307Z Running tests...
2023-01-11T22:03:10.0762504Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0762748Z   test_graph_for_python_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15514
2023-01-11T22:03:10.0762909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15515
2023-01-11T22:03:10.0763063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15516
2023-01-11T22:03:10.0763204Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15517
2023-01-11T22:03:10.0763477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0763601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0763884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0764054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0764241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphp9wlr4
2023-01-11T22:03:10.0764466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphp9wlr4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0764741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0764866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0765135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0765267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0765451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpua76ga_4
2023-01-11T22:03:10.0765650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpua76ga_4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0765924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0766050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0766329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0766463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0766635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz6s6wp66
2023-01-11T22:03:10.0766829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz6s6wp66/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0767098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0767223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0767507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0767646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0767829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59fa3tiv
2023-01-11T22:03:10.0768021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59fa3tiv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0768184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0768336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0768498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0768658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0768838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0769016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0769317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0769493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0769661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0769955Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0770244Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0770532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0770630Z ok (2.949s)
2023-01-11T22:03:10.0770636Z 
2023-01-11T22:03:10.0770917Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0770996Z Ran 1 test in 2.950s
2023-01-11T22:03:10.0771002Z 
2023-01-11T22:03:10.0771067Z OK
2023-01-11T22:03:10.0771072Z 
2023-01-11T22:03:10.0771158Z Generating XML reports...
2023-01-11T22:03:10.0771532Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213243.xml
2023-01-11T22:03:10.0771808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0771920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0772203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0772342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0772530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50z4nx96
2023-01-11T22:03:10.0772729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50z4nx96/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0772986Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0772991Z 
2023-01-11T22:03:10.0773065Z Running tests...
2023-01-11T22:03:10.0773263Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0773500Z   test_graph_for_python_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15749
2023-01-11T22:03:10.0773657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15750
2023-01-11T22:03:10.0773814Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15751
2023-01-11T22:03:10.0773970Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15752
2023-01-11T22:03:10.0774250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0774374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0774655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0774789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0774974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2fpyski
2023-01-11T22:03:10.0775157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2fpyski/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0775431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0775558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0775841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0775977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0776164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9zr38vb
2023-01-11T22:03:10.0776436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0776559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0776826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0776959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0777143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbmzc06p
2023-01-11T22:03:10.0777368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9zr38vb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0777584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbmzc06p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0777857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0777981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0778260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0778396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0778567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8gkginz
2023-01-11T22:03:10.0778759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8gkginz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0778926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0779092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0779254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0779423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0779599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0779774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0779950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0780109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0780411Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0780711Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0781001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0781292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0781361Z ok (2.943s)
2023-01-11T22:03:10.0781366Z 
2023-01-11T22:03:10.0781563Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0781637Z Ran 1 test in 2.943s
2023-01-11T22:03:10.0781644Z 
2023-01-11T22:03:10.0781804Z OK
2023-01-11T22:03:10.0781823Z 
2023-01-11T22:03:10.0781895Z Generating XML reports...
2023-01-11T22:03:10.0782268Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213248.xml
2023-01-11T22:03:10.0782548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0782674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0782959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0783096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0783286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8fv0fwa
2023-01-11T22:03:10.0783483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8fv0fwa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0783729Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0783748Z 
2023-01-11T22:03:10.0783866Z Running tests...
2023-01-11T22:03:10.0784063Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0784341Z   test_mixed_requires_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15984
2023-01-11T22:03:10.0784502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15985
2023-01-11T22:03:10.0784658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15986
2023-01-11T22:03:10.0784813Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15987
2023-01-11T22:03:10.0785090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0785215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0785485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0785624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0785812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5urzuwdp
2023-01-11T22:03:10.0786013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5urzuwdp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0786287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0786412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0786696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0786830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0787025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdxjhtyi
2023-01-11T22:03:10.0787391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0787521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0787807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0787948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0788133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1otg8ud
2023-01-11T22:03:10.0788333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdxjhtyi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0788526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1otg8ud/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0788798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0788909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0789194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0789330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0789518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqnfhug68
2023-01-11T22:03:10.0789714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqnfhug68/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0789881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0790048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0790212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0790362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0790431Z ok (2.743s)
2023-01-11T22:03:10.0790474Z 
2023-01-11T22:03:10.0790674Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0790752Z Ran 1 test in 2.744s
2023-01-11T22:03:10.0790758Z 
2023-01-11T22:03:10.0790848Z OK
2023-01-11T22:03:10.0790853Z 
2023-01-11T22:03:10.0790937Z Generating XML reports...
2023-01-11T22:03:10.0791309Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213253.xml
2023-01-11T22:03:10.0791581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0791704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0791971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0792109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0792292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmvsmcly
2023-01-11T22:03:10.0792489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmvsmcly/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0792744Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0792751Z 
2023-01-11T22:03:10.0792824Z Running tests...
2023-01-11T22:03:10.0793019Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0793258Z   test_multiple_backward (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16227
2023-01-11T22:03:10.0793414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16228
2023-01-11T22:03:10.0793557Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16229
2023-01-11T22:03:10.0793709Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16230
2023-01-11T22:03:10.0793987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0794112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0794394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0794531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0794715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcqb_i1oy
2023-01-11T22:03:10.0794911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcqb_i1oy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0795171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0795293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0795572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0795709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0795892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxw_i6dbr
2023-01-11T22:03:10.0796085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxw_i6dbr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0796357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0796479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0796759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0796880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0797064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5sycvu_
2023-01-11T22:03:10.0797291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5sycvu_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0797589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0797712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0797992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0798126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0798309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu4roj0zn
2023-01-11T22:03:10.0798489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu4roj0zn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0798656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0798918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0799087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0799250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0799319Z ok (3.347s)
2023-01-11T22:03:10.0799324Z 
2023-01-11T22:03:10.0799524Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0799601Z Ran 1 test in 3.347s
2023-01-11T22:03:10.0799606Z 
2023-01-11T22:03:10.0799667Z OK
2023-01-11T22:03:10.0799672Z 
2023-01-11T22:03:10.0799743Z Generating XML reports...
2023-01-11T22:03:10.0800114Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213258.xml
2023-01-11T22:03:10.0800392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0800520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0800805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0800942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0801126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9eua1ph5
2023-01-11T22:03:10.0801318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9eua1ph5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0801561Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0801581Z 
2023-01-11T22:03:10.0801640Z Running tests...
2023-01-11T22:03:10.0801836Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0802088Z   test_multiple_backward_with_errors (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16470
2023-01-11T22:03:10.0802246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16471
2023-01-11T22:03:10.0802406Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16472
2023-01-11T22:03:10.0802562Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16473
2023-01-11T22:03:10.0802839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0802961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0803230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0803365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0803549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3wx2v8el
2023-01-11T22:03:10.0803785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3wx2v8el/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0804096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0804223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0804504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0804639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0804811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6d66u0r
2023-01-11T22:03:10.0805004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6d66u0r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0805276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0805401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0805684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0805818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0806001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7k3wctb
2023-01-11T22:03:10.0806196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7k3wctb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0806467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0806578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0806856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0806990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0807175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigk53kbi
2023-01-11T22:03:10.0807370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigk53kbi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0807535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0807701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0807866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0808030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0808198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0808376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0808555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0808860Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0809033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0809335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0809634Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0809930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0810222Z [W tensorpipe_agent.cpp:624] RPC agent for worker3 won't send response to request #150 to worker2, as the agent is shutting down
2023-01-11T22:03:10.0810536Z [W tensorpipe_agent.cpp:624] RPC agent for worker0 won't send response to request #151 to worker1, as the agent is shutting down
2023-01-11T22:03:10.0810854Z [W tensorpipe_agent.cpp:624] RPC agent for worker0 won't send response to request #150 to worker3, as the agent is shutting down
2023-01-11T22:03:10.0811127Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0811394Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0811653Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0811914Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0812139Z [E container.cpp:256] Could not release Dist Autograd Context on node 3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0812408Z [W tensorpipe_agent.cpp:939] RPC agent for worker1 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0812633Z [E container.cpp:256] Could not release Dist Autograd Context on node 0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0812899Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.0813117Z [E container.cpp:256] Could not release Dist Autograd Context on node 2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0813381Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0813624Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0813844Z [E container.cpp:256] Could not release Dist Autograd Context on node 0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0814097Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0814353Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0814618Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.0814906Z [W tensorpipe_agent.cpp:624] RPC agent for worker2 won't send response to request #150 to worker1, as the agent is shutting down
2023-01-11T22:03:10.0814978Z ok (2.954s)
2023-01-11T22:03:10.0814984Z 
2023-01-11T22:03:10.0815184Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0836530Z Ran 1 test in 2.955s
2023-01-11T22:03:10.0836552Z 
2023-01-11T22:03:10.0836686Z OK
2023-01-11T22:03:10.0836692Z 
2023-01-11T22:03:10.0836821Z Generating XML reports...
2023-01-11T22:03:10.0837268Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213303.xml
2023-01-11T22:03:10.0837728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0837941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0838234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0838373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0838566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv99fp4qe
2023-01-11T22:03:10.0838766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv99fp4qe/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0839123Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0839130Z 
2023-01-11T22:03:10.0839207Z Running tests...
2023-01-11T22:03:10.0839395Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0839658Z   test_nested_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16725
2023-01-11T22:03:10.0839821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16726
2023-01-11T22:03:10.0839980Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16727
2023-01-11T22:03:10.0840137Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16728
2023-01-11T22:03:10.0840420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0840550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0840837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0840976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0841154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph82v57h3
2023-01-11T22:03:10.0841354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph82v57h3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0841630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0841756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0842039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0842176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0842362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwy74hj_5
2023-01-11T22:03:10.0842558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwy74hj_5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0842835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0842946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0843230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0843366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0843551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_apr4wz
2023-01-11T22:03:10.0843745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_apr4wz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0844018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0844141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0844419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0844577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0844791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz3h7krn2
2023-01-11T22:03:10.0844987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz3h7krn2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0845154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0845319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0845482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0845647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0845714Z ok (2.643s)
2023-01-11T22:03:10.0845720Z 
2023-01-11T22:03:10.0845920Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0845987Z Ran 1 test in 2.644s
2023-01-11T22:03:10.0845992Z 
2023-01-11T22:03:10.0846052Z OK
2023-01-11T22:03:10.0846058Z 
2023-01-11T22:03:10.0846144Z Generating XML reports...
2023-01-11T22:03:10.0846521Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213308.xml
2023-01-11T22:03:10.0846796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0846922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0847205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0847340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0847510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wh8mw4c
2023-01-11T22:03:10.0847707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wh8mw4c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0847962Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0847968Z 
2023-01-11T22:03:10.0848043Z Running tests...
2023-01-11T22:03:10.0848243Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0848480Z   test_nested_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16968
2023-01-11T22:03:10.0848638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16969
2023-01-11T22:03:10.0848794Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16970
2023-01-11T22:03:10.0848948Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16971
2023-01-11T22:03:10.0849210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0849337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0849627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0849765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0849952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjkuzmvnl
2023-01-11T22:03:10.0850149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjkuzmvnl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0850423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0850548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0850816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0850982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0851196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2z6uerbi
2023-01-11T22:03:10.0851394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2z6uerbi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0851671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0851796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0852077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0852212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0852398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68uka64g
2023-01-11T22:03:10.0852578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68uka64g/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0852854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0852978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0853259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0853394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0853580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp793uma1h
2023-01-11T22:03:10.0853775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp793uma1h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0853940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0854104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0854257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0854422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0854490Z ok (2.640s)
2023-01-11T22:03:10.0854497Z 
2023-01-11T22:03:10.0854695Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0854772Z Ran 1 test in 2.640s
2023-01-11T22:03:10.0854778Z 
2023-01-11T22:03:10.0854838Z OK
2023-01-11T22:03:10.0854844Z 
2023-01-11T22:03:10.0854928Z Generating XML reports...
2023-01-11T22:03:10.0855305Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213312.xml
2023-01-11T22:03:10.0855570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0855695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0855978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0856118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0856304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhxdcrp_
2023-01-11T22:03:10.0856502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhxdcrp_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0856760Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0856766Z 
2023-01-11T22:03:10.0856840Z Running tests...
2023-01-11T22:03:10.0857037Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0857161Z   test_no_grad_copy (__main__.TensorPipeDistAutogradTest)
2023-01-11T22:03:10.0857362Z Similar to test in test_autograd.py. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17191
2023-01-11T22:03:10.0857550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17192
2023-01-11T22:03:10.0857735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17193
2023-01-11T22:03:10.0857890Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17194
2023-01-11T22:03:10.0858167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0858294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0858577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0858699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0858885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbejhqe2t
2023-01-11T22:03:10.0859081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbejhqe2t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0859359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0859484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0859769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0859907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0860094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppbg54sra
2023-01-11T22:03:10.0860290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppbg54sra/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0860549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0860676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0860961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0861099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0861282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgt7eulc
2023-01-11T22:03:10.0861477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgt7eulc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0861886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0862013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0862279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0862414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0862602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kqxbofq
2023-01-11T22:03:10.0862800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kqxbofq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0862967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0863133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0863298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0863462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0863530Z ok (2.644s)
2023-01-11T22:03:10.0863536Z 
2023-01-11T22:03:10.0863722Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0863800Z Ran 1 test in 2.644s
2023-01-11T22:03:10.0863805Z 
2023-01-11T22:03:10.0863868Z OK
2023-01-11T22:03:10.0863919Z 
2023-01-11T22:03:10.0864005Z Generating XML reports...
2023-01-11T22:03:10.0864416Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213317.xml
2023-01-11T22:03:10.0864697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0864824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0865109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0865232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0865420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoye08ahi
2023-01-11T22:03:10.0865616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoye08ahi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0865875Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0865884Z 
2023-01-11T22:03:10.0865959Z Running tests...
2023-01-11T22:03:10.0866160Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0866404Z   test_no_grad_copy_sparse (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17434
2023-01-11T22:03:10.0866564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17435
2023-01-11T22:03:10.0866723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17436
2023-01-11T22:03:10.0866866Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17437
2023-01-11T22:03:10.0867138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0867265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0867551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0867705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0867978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkljxbly2
2023-01-11T22:03:10.0868199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkljxbly2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0868480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0868591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0868873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0869009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0869196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzz4a2ey
2023-01-11T22:03:10.0869394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzz4a2ey/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0869672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0869800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0870082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0870221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0870394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvegnkeun
2023-01-11T22:03:10.0870595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvegnkeun/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0870871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0871032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0871350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0871488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0871674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ymhhdj2
2023-01-11T22:03:10.0871868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ymhhdj2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0872021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0872185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0872350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0872514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0873014Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2023-01-11T22:03:10.0873154Z   "Argument order of nn.functional.embedding_bag was changed. "
2023-01-11T22:03:10.0873647Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2023-01-11T22:03:10.0873784Z   "Argument order of nn.functional.embedding_bag was changed. "
2023-01-11T22:03:10.0874268Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2023-01-11T22:03:10.0874410Z   "Argument order of nn.functional.embedding_bag was changed. "
2023-01-11T22:03:10.0874895Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2326: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2023-01-11T22:03:10.0875018Z   "Argument order of nn.functional.embedding_bag was changed. "
2023-01-11T22:03:10.0875086Z ok (2.655s)
2023-01-11T22:03:10.0875092Z 
2023-01-11T22:03:10.0875293Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0875370Z Ran 1 test in 2.655s
2023-01-11T22:03:10.0875376Z 
2023-01-11T22:03:10.0875437Z OK
2023-01-11T22:03:10.0875442Z 
2023-01-11T22:03:10.0875526Z Generating XML reports...
2023-01-11T22:03:10.0875896Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213322.xml
2023-01-11T22:03:10.0876177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0876302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0876572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0876709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0876899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppro2h311
2023-01-11T22:03:10.0877095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppro2h311/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0877351Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0877357Z 
2023-01-11T22:03:10.0877431Z Running tests...
2023-01-11T22:03:10.0877668Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0877956Z   test_no_graph_with_tensors_not_require_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17689
2023-01-11T22:03:10.0878103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17690
2023-01-11T22:03:10.0878259Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17691
2023-01-11T22:03:10.0878413Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17692
2023-01-11T22:03:10.0878878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0879032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0879326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0879468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0879659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjtes6q8
2023-01-11T22:03:10.0879858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjtes6q8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0880117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0880243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0880527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0880664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0880849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekrtucmg
2023-01-11T22:03:10.0881048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekrtucmg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0881326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0881451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0881731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0881852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0882038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzz6qtmq3
2023-01-11T22:03:10.0882233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzz6qtmq3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0882503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0882627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0882914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0883052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0883236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43o9ua0s
2023-01-11T22:03:10.0883416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43o9ua0s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0883580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0883745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0883908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0884069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0884295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0884517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0884694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0884868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0885160Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0885458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0885754Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0886046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0886117Z ok (2.849s)
2023-01-11T22:03:10.0886123Z 
2023-01-11T22:03:10.0886327Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0886402Z Ran 1 test in 2.849s
2023-01-11T22:03:10.0886408Z 
2023-01-11T22:03:10.0886468Z OK
2023-01-11T22:03:10.0886473Z 
2023-01-11T22:03:10.0886557Z Generating XML reports...
2023-01-11T22:03:10.0886916Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213326.xml
2023-01-11T22:03:10.0887188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0887316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0887599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0887740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0887927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo09mrez_
2023-01-11T22:03:10.0888126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo09mrez_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0888382Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0888387Z 
2023-01-11T22:03:10.0888459Z Running tests...
2023-01-11T22:03:10.0888642Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0888910Z   test_no_graph_with_tensors_not_require_grad_remote (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17924
2023-01-11T22:03:10.0889067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17925
2023-01-11T22:03:10.0889223Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17926
2023-01-11T22:03:10.0889379Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17927
2023-01-11T22:03:10.0889657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0889782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0890065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0890189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0890374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tzn897n
2023-01-11T22:03:10.0890569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tzn897n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0890844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0891001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0891321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0891459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0891646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5nu141o
2023-01-11T22:03:10.0891842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5nu141o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0892102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0892227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0892507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0892644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0892833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprq8npdsi
2023-01-11T22:03:10.0893030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprq8npdsi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0893303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0893429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0893696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0893830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0894015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkv5ihaiq
2023-01-11T22:03:10.0894210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkv5ihaiq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0894379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0894546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0894708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0894870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0895048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.0895212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.0895385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.0895683Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0895983Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0896159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.0896462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0896759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.0896828Z ok (2.845s)
2023-01-11T22:03:10.0896834Z 
2023-01-11T22:03:10.0897034Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0897097Z Ran 1 test in 2.845s
2023-01-11T22:03:10.0897102Z 
2023-01-11T22:03:10.0897163Z OK
2023-01-11T22:03:10.0897168Z 
2023-01-11T22:03:10.0897252Z Generating XML reports...
2023-01-11T22:03:10.0897624Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213331.xml
2023-01-11T22:03:10.0897996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0898123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0898408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0898545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0898719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgc5po7nl
2023-01-11T22:03:10.0898917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgc5po7nl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0899172Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0899177Z 
2023-01-11T22:03:10.0899255Z Running tests...
2023-01-11T22:03:10.0899454Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0899689Z   test_post_hooks (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18159
2023-01-11T22:03:10.0899850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18160
2023-01-11T22:03:10.0900008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18161
2023-01-11T22:03:10.0900165Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18162
2023-01-11T22:03:10.0900429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0900554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0900836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0900977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0901166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplaecv98a
2023-01-11T22:03:10.0901367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplaecv98a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0901644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0901943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0902259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0902383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0902570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2hw9hu3_
2023-01-11T22:03:10.0902768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2hw9hu3_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0903047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0903176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0903458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0903595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0903782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2wy_psd
2023-01-11T22:03:10.0903965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2wy_psd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0904240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0904366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0904712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0904883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0905074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeytkyqvw
2023-01-11T22:03:10.0905272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeytkyqvw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0905439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0905605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0905756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0905919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0905991Z ok (2.637s)
2023-01-11T22:03:10.0905998Z 
2023-01-11T22:03:10.0906200Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0906278Z Ran 1 test in 2.637s
2023-01-11T22:03:10.0906286Z 
2023-01-11T22:03:10.0906350Z OK
2023-01-11T22:03:10.0906355Z 
2023-01-11T22:03:10.0906441Z Generating XML reports...
2023-01-11T22:03:10.0906812Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213336.xml
2023-01-11T22:03:10.0907073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0907199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0907479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0907616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0907805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptxjye7gv
2023-01-11T22:03:10.0908003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptxjye7gv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0908262Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0908267Z 
2023-01-11T22:03:10.0908342Z Running tests...
2023-01-11T22:03:10.0908539Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0908768Z   test_remote_complex_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18402
2023-01-11T22:03:10.0908925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18403
2023-01-11T22:03:10.0909080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18404
2023-01-11T22:03:10.0909234Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18405
2023-01-11T22:03:10.0909514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0909640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0909922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0910058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0910233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckimfmmw
2023-01-11T22:03:10.0910432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckimfmmw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0910706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0910830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0911145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0911281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0911493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rruj4f7
2023-01-11T22:03:10.0911689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rruj4f7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0911960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0912071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0912350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0912484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0912670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4f2fmz9k
2023-01-11T22:03:10.0912868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4f2fmz9k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0913145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0913268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0913546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0913680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0913851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_lma5m3
2023-01-11T22:03:10.0914043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_lma5m3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0914209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0914375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0914541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0914704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0914772Z ok (2.738s)
2023-01-11T22:03:10.0914777Z 
2023-01-11T22:03:10.0914974Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0915037Z Ran 1 test in 2.738s
2023-01-11T22:03:10.0915042Z 
2023-01-11T22:03:10.0915104Z OK
2023-01-11T22:03:10.0915108Z 
2023-01-11T22:03:10.0915193Z Generating XML reports...
2023-01-11T22:03:10.0915563Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213340.xml
2023-01-11T22:03:10.0915836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0915964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0916247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0916384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0916557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmn5ra139
2023-01-11T22:03:10.0916752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmn5ra139/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0917008Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0917013Z 
2023-01-11T22:03:10.0917091Z Running tests...
2023-01-11T22:03:10.0917290Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0917527Z   test_rpc_complex_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18625
2023-01-11T22:03:10.0917716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18626
2023-01-11T22:03:10.0917900Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18627
2023-01-11T22:03:10.0918056Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18628
2023-01-11T22:03:10.0918317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0918441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0918722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0918932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0919122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskgqq30u
2023-01-11T22:03:10.0919324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskgqq30u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0919607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0919734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0920016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0920139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0920322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qq_dlr0
2023-01-11T22:03:10.0920517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qq_dlr0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0920789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0920918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0921203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0921338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0921527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp0y19ezi
2023-01-11T22:03:10.0921708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp0y19ezi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0921985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0922110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0922391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0922529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0922719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2b_kkj75
2023-01-11T22:03:10.0922915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2b_kkj75/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0923081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0923244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0923394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0923555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0923623Z ok (2.645s)
2023-01-11T22:03:10.0923629Z 
2023-01-11T22:03:10.0923826Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0923902Z Ran 1 test in 2.645s
2023-01-11T22:03:10.0923942Z 
2023-01-11T22:03:10.0924003Z OK
2023-01-11T22:03:10.0924009Z 
2023-01-11T22:03:10.0924097Z Generating XML reports...
2023-01-11T22:03:10.0924494Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213345.xml
2023-01-11T22:03:10.0924756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0924881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0925162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0925297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0925484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1yfwoo9
2023-01-11T22:03:10.0925678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1yfwoo9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0925939Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0925945Z 
2023-01-11T22:03:10.0926018Z Running tests...
2023-01-11T22:03:10.0926217Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0926447Z   test_thread_local_context_id (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18848
2023-01-11T22:03:10.0926606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18849
2023-01-11T22:03:10.0926761Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18850
2023-01-11T22:03:10.0926917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18851
2023-01-11T22:03:10.0927194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0927320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0927609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0927749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0927922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4k4kubqm
2023-01-11T22:03:10.0928121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4k4kubqm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0928396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0928521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0928802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0928939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0929128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjma49kau
2023-01-11T22:03:10.0929407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0929531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0929801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0929936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0930120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ahwf5jx
2023-01-11T22:03:10.0930317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjma49kau/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0930507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ahwf5jx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0930781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0930936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0931262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0931397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0931568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbzc5xusm
2023-01-11T22:03:10.0931762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbzc5xusm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0931926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0932091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0932254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0932419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0932488Z ok (3.647s)
2023-01-11T22:03:10.0932495Z 
2023-01-11T22:03:10.0932692Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0932754Z Ran 1 test in 3.647s
2023-01-11T22:03:10.0932760Z 
2023-01-11T22:03:10.0932821Z OK
2023-01-11T22:03:10.0932826Z 
2023-01-11T22:03:10.0932909Z Generating XML reports...
2023-01-11T22:03:10.0933281Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213349.xml
2023-01-11T22:03:10.0933553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0933679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0933963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0934100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0934288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpmbjkpc
2023-01-11T22:03:10.0934472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpmbjkpc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0934725Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0934731Z 
2023-01-11T22:03:10.0934804Z Running tests...
2023-01-11T22:03:10.0935003Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0935235Z   test_trainer_ps (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19091
2023-01-11T22:03:10.0935393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19092
2023-01-11T22:03:10.0935553Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19093
2023-01-11T22:03:10.0935712Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19094
2023-01-11T22:03:10.0935974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0936098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0936379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0936514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0936701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphv36yusu
2023-01-11T22:03:10.0936896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphv36yusu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0937172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0937327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0937643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0937768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0937954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk42xenop
2023-01-11T22:03:10.0938148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk42xenop/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0938423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0938546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0938826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0938961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0939146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphyy2dfju
2023-01-11T22:03:10.0939329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphyy2dfju/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0939601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0939725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0940005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0940140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0940327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvu_q_njq
2023-01-11T22:03:10.0940518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvu_q_njq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0940686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0940855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0941005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0941168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0941236Z ok (2.851s)
2023-01-11T22:03:10.0941242Z 
2023-01-11T22:03:10.0941440Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0941515Z Ran 1 test in 2.851s
2023-01-11T22:03:10.0941521Z 
2023-01-11T22:03:10.0941582Z OK
2023-01-11T22:03:10.0941588Z 
2023-01-11T22:03:10.0941808Z Generating XML reports...
2023-01-11T22:03:10.0942244Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213355.xml
2023-01-11T22:03:10.0942506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0942636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0942918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0943055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0943242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmponai2d0p
2023-01-11T22:03:10.0943439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmponai2d0p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0943694Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0943700Z 
2023-01-11T22:03:10.0943774Z Running tests...
2023-01-11T22:03:10.0944028Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0944307Z   test_trainer_ps_torchscript_functions (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19334
2023-01-11T22:03:10.0944470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19335
2023-01-11T22:03:10.0944627Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19336
2023-01-11T22:03:10.0944783Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19337
2023-01-11T22:03:10.0945064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0945190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0945472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0945610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0945798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp04_d5zpp
2023-01-11T22:03:10.0945983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp04_d5zpp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0946257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0946381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0946661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0946796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0946979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzern6gch
2023-01-11T22:03:10.0947173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzern6gch/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0947447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0947560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0947842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0947976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0948163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7vf8xtw
2023-01-11T22:03:10.0948359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7vf8xtw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0948635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0948761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0949044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0949179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0949353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptee9l_z
2023-01-11T22:03:10.0949548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptee9l_z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0949716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0949881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0950046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0950209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0950277Z ok (2.843s)
2023-01-11T22:03:10.0950312Z 
2023-01-11T22:03:10.0950514Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0950579Z Ran 1 test in 2.844s
2023-01-11T22:03:10.0950585Z 
2023-01-11T22:03:10.0950647Z OK
2023-01-11T22:03:10.0950689Z 
2023-01-11T22:03:10.0950774Z Generating XML reports...
2023-01-11T22:03:10.0951145Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213400.xml
2023-01-11T22:03:10.0951417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0951543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0951823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0951958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0952141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35bki8yq
2023-01-11T22:03:10.0952323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35bki8yq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0952577Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0952583Z 
2023-01-11T22:03:10.0952656Z Running tests...
2023-01-11T22:03:10.0952852Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0953096Z   test_worker_ids_recorded (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19577
2023-01-11T22:03:10.0953255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19578
2023-01-11T22:03:10.0953410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19579
2023-01-11T22:03:10.0953564Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19580
2023-01-11T22:03:10.0953830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0953954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0954256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0954396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0954579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbm9w3yuy
2023-01-11T22:03:10.0954776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbm9w3yuy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0955047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0955171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0955449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0955573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0955759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8yct6hw
2023-01-11T22:03:10.0955953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8yct6hw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0956225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0956349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0956629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0956763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0956947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5vwlofx
2023-01-11T22:03:10.0957173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5vwlofx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0957466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0957590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0957869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0958004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0958188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdu2b_2_1
2023-01-11T22:03:10.0958380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdu2b_2_1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0958545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0958713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0958931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0959100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0959171Z ok (2.748s)
2023-01-11T22:03:10.0959176Z 
2023-01-11T22:03:10.0959374Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0959451Z Ran 1 test in 2.748s
2023-01-11T22:03:10.0959456Z 
2023-01-11T22:03:10.0959519Z OK
2023-01-11T22:03:10.0959523Z 
2023-01-11T22:03:10.0959609Z Generating XML reports...
2023-01-11T22:03:10.0959983Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213405.xml
2023-01-11T22:03:10.0960259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0960373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0960658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0960794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0960980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqql4grvk
2023-01-11T22:03:10.0961176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqql4grvk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0961434Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0961439Z 
2023-01-11T22:03:10.0961514Z Running tests...
2023-01-11T22:03:10.0961710Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0962406Z   test_dist_optim (__main__.TensorPipeDistOptimizerTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/72997 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.318s)
2023-01-11T22:03:10.0962415Z 
2023-01-11T22:03:10.0962610Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0962672Z Ran 1 test in 0.318s
2023-01-11T22:03:10.0962677Z 
2023-01-11T22:03:10.0962748Z OK (skipped=1)
2023-01-11T22:03:10.0962753Z 
2023-01-11T22:03:10.0962839Z Generating XML reports...
2023-01-11T22:03:10.0963215Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20230111213409.xml
2023-01-11T22:03:10.0963495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0963621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0963944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0964114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0964291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptiqjlehr
2023-01-11T22:03:10.0964490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptiqjlehr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0964747Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0964752Z 
2023-01-11T22:03:10.0964827Z Running tests...
2023-01-11T22:03:10.0965026Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0965271Z   test_dist_optim_exception (__main__.TensorPipeDistOptimizerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19810
2023-01-11T22:03:10.0965430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19811
2023-01-11T22:03:10.0965589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19812
2023-01-11T22:03:10.0965746Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19813
2023-01-11T22:03:10.0966010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0966133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0966416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0966551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0966738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76o4fkor
2023-01-11T22:03:10.0966935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76o4fkor/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0967210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0967336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0967603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0967736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0967920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzogjnxfw
2023-01-11T22:03:10.0968118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzogjnxfw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0968390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0968514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0968793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0968933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0969120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5q6uhik
2023-01-11T22:03:10.0969302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5q6uhik/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0969573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0969697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0969984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0970176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0970381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlz4x4fv
2023-01-11T22:03:10.0970618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlz4x4fv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0970818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.0970972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.0971135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.0971300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.0972100Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.FailingOptimizer'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2023-01-11T22:03:10.0972198Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.0972338Z ValueError('Error running optimizer.')
2023-01-11T22:03:10.0972433Z Traceback (most recent call last):
2023-01-11T22:03:10.0972698Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.0972835Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.0973119Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 91, in _local_optimizer_step
2023-01-11T22:03:10.0973203Z     local_optim.step(autograd_ctx_id)
2023-01-11T22:03:10.0973456Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 82, in step
2023-01-11T22:03:10.0973536Z     self.optim.step()
2023-01-11T22:03:10.0973774Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 219, in wrapper
2023-01-11T22:03:10.0973860Z     out = func(*args, **kwargs)
2023-01-11T22:03:10.0974152Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2023-01-11T22:03:10.0974267Z     raise ValueError("Error running optimizer.")
2023-01-11T22:03:10.0974353Z ValueError: Error running optimizer.
2023-01-11T22:03:10.0974371Z 
2023-01-11T22:03:10.0975149Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.FailingOptimizer'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2023-01-11T22:03:10.0975242Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.0975379Z ValueError('Error running optimizer.')
2023-01-11T22:03:10.0975472Z Traceback (most recent call last):
2023-01-11T22:03:10.0975735Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.0975872Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.0976155Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 91, in _local_optimizer_step
2023-01-11T22:03:10.0976252Z     local_optim.step(autograd_ctx_id)
2023-01-11T22:03:10.0976505Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 82, in step
2023-01-11T22:03:10.0976575Z     self.optim.step()
2023-01-11T22:03:10.0976813Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 219, in wrapper
2023-01-11T22:03:10.0976897Z     out = func(*args, **kwargs)
2023-01-11T22:03:10.0977189Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2023-01-11T22:03:10.0977305Z     raise ValueError("Error running optimizer.")
2023-01-11T22:03:10.0977438Z ValueError: Error running optimizer.
2023-01-11T22:03:10.0977443Z 
2023-01-11T22:03:10.0978264Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.FailingOptimizer'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2023-01-11T22:03:10.0979039Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.FailingOptimizer'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2023-01-11T22:03:10.0979135Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.0979273Z ValueError('Error running optimizer.')
2023-01-11T22:03:10.0979353Z Traceback (most recent call last):
2023-01-11T22:03:10.0979617Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.0979752Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.0980035Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 91, in _local_optimizer_step
2023-01-11T22:03:10.0980132Z     local_optim.step(autograd_ctx_id)
2023-01-11T22:03:10.0980387Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 82, in step
2023-01-11T22:03:10.0980472Z     self.optim.step()
2023-01-11T22:03:10.0980708Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 219, in wrapper
2023-01-11T22:03:10.0980776Z     out = func(*args, **kwargs)
2023-01-11T22:03:10.0981070Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2023-01-11T22:03:10.0981185Z     raise ValueError("Error running optimizer.")
2023-01-11T22:03:10.0981283Z ValueError: Error running optimizer.
2023-01-11T22:03:10.0981289Z 
2023-01-11T22:03:10.0981381Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.0981518Z ValueError('Error running optimizer.')
2023-01-11T22:03:10.0981612Z Traceback (most recent call last):
2023-01-11T22:03:10.0982026Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.0982148Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.0982428Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 91, in _local_optimizer_step
2023-01-11T22:03:10.0982524Z     local_optim.step(autograd_ctx_id)
2023-01-11T22:03:10.0982779Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 82, in step
2023-01-11T22:03:10.0982865Z     self.optim.step()
2023-01-11T22:03:10.0983103Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 219, in wrapper
2023-01-11T22:03:10.0983188Z     out = func(*args, **kwargs)
2023-01-11T22:03:10.0983480Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2023-01-11T22:03:10.0983580Z     raise ValueError("Error running optimizer.")
2023-01-11T22:03:10.0983679Z ValueError: Error running optimizer.
2023-01-11T22:03:10.0983684Z 
2023-01-11T22:03:10.0983776Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.0983914Z ValueError('Error running optimizer.')
2023-01-11T22:03:10.0984011Z Traceback (most recent call last):
2023-01-11T22:03:10.0984270Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.0984406Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.0984765Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 91, in _local_optimizer_step
2023-01-11T22:03:10.0984886Z     local_optim.step(autograd_ctx_id)
2023-01-11T22:03:10.0985141Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 82, in step
2023-01-11T22:03:10.0985226Z     self.optim.step()
2023-01-11T22:03:10.0985465Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 219, in wrapper
2023-01-11T22:03:10.0985549Z     out = func(*args, **kwargs)
2023-01-11T22:03:10.0985842Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2023-01-11T22:03:10.0985957Z     raise ValueError("Error running optimizer.")
2023-01-11T22:03:10.0986042Z ValueError: Error running optimizer.
2023-01-11T22:03:10.0986062Z 
2023-01-11T22:03:10.0986140Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.0986280Z ValueError('Error running optimizer.')
2023-01-11T22:03:10.0986374Z Traceback (most recent call last):
2023-01-11T22:03:10.0986636Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.0986771Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.0987050Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 91, in _local_optimizer_step
2023-01-11T22:03:10.0987146Z     local_optim.step(autograd_ctx_id)
2023-01-11T22:03:10.0987387Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 82, in step
2023-01-11T22:03:10.0987467Z     self.optim.step()
2023-01-11T22:03:10.0987702Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 219, in wrapper
2023-01-11T22:03:10.0987785Z     out = func(*args, **kwargs)
2023-01-11T22:03:10.0988078Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2023-01-11T22:03:10.0988197Z     raise ValueError("Error running optimizer.")
2023-01-11T22:03:10.0988296Z ValueError: Error running optimizer.
2023-01-11T22:03:10.0988300Z 
2023-01-11T22:03:10.0988391Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.0988514Z ValueError('Error running optimizer.')
2023-01-11T22:03:10.0988606Z Traceback (most recent call last):
2023-01-11T22:03:10.0988864Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.0988998Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.0989276Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 91, in _local_optimizer_step
2023-01-11T22:03:10.0989376Z     local_optim.step(autograd_ctx_id)
2023-01-11T22:03:10.0989630Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 82, in step
2023-01-11T22:03:10.0989714Z     self.optim.step()
2023-01-11T22:03:10.0989937Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 219, in wrapper
2023-01-11T22:03:10.0990023Z     out = func(*args, **kwargs)
2023-01-11T22:03:10.0990314Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2023-01-11T22:03:10.0990428Z     raise ValueError("Error running optimizer.")
2023-01-11T22:03:10.0990525Z ValueError: Error running optimizer.
2023-01-11T22:03:10.0990530Z 
2023-01-11T22:03:10.0990624Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.0990760Z ValueError('Error running optimizer.')
2023-01-11T22:03:10.0990854Z Traceback (most recent call last):
2023-01-11T22:03:10.0991101Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.0991233Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.0991545Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 91, in _local_optimizer_step
2023-01-11T22:03:10.0991668Z     local_optim.step(autograd_ctx_id)
2023-01-11T22:03:10.0991921Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 82, in step
2023-01-11T22:03:10.0992001Z     self.optim.step()
2023-01-11T22:03:10.0992234Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 219, in wrapper
2023-01-11T22:03:10.0992318Z     out = func(*args, **kwargs)
2023-01-11T22:03:10.0992600Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2023-01-11T22:03:10.0992713Z     raise ValueError("Error running optimizer.")
2023-01-11T22:03:10.0992811Z ValueError: Error running optimizer.
2023-01-11T22:03:10.0992817Z 
2023-01-11T22:03:10.0992884Z ok (2.737s)
2023-01-11T22:03:10.0992890Z 
2023-01-11T22:03:10.0993093Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0993168Z Ran 1 test in 2.737s
2023-01-11T22:03:10.0993174Z 
2023-01-11T22:03:10.0993235Z OK
2023-01-11T22:03:10.0993240Z 
2023-01-11T22:03:10.0993325Z Generating XML reports...
2023-01-11T22:03:10.0993692Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20230111213411.xml
2023-01-11T22:03:10.0993968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0994093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0994378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0994515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0994705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbe56oua1
2023-01-11T22:03:10.0994905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbe56oua1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0995161Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.0995167Z 
2023-01-11T22:03:10.0995241Z Running tests...
2023-01-11T22:03:10.0995428Z ----------------------------------------------------------------------
2023-01-11T22:03:10.0995689Z   test_dist_optim_exception_on_constructor (__main__.TensorPipeDistOptimizerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20053
2023-01-11T22:03:10.0995848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20054
2023-01-11T22:03:10.0996003Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20055
2023-01-11T22:03:10.0996157Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20056
2023-01-11T22:03:10.0996435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0996561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0996851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0996975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0997162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4c3do40
2023-01-11T22:03:10.0997358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4c3do40/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0997636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0997762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0998044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0998209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0998458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpete7eupt
2023-01-11T22:03:10.0998657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpete7eupt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.0999000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.0999129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.0999412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.0999549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.0999736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbheg6wao
2023-01-11T22:03:10.0999937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbheg6wao/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1000214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1000339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1000607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1000744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1000929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyyu27k7
2023-01-11T22:03:10.1001128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyyu27k7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1001298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1001464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1001630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1001795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1002614Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.OptimizerFailingOnConstructor'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2023-01-11T22:03:10.1002710Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.1002838Z ValueError('Error creating optimizer.')
2023-01-11T22:03:10.1002932Z Traceback (most recent call last):
2023-01-11T22:03:10.1003194Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1003332Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1003613Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 86, in _new_local_optimizer
2023-01-11T22:03:10.1003765Z     return rpc.RRef(_LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2023-01-11T22:03:10.1004024Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 74, in __init__
2023-01-11T22:03:10.1004149Z     self.optim = optim_cls(self._local_params, *args, **kwargs)
2023-01-11T22:03:10.1004435Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2023-01-11T22:03:10.1004550Z     raise ValueError("Error creating optimizer.")
2023-01-11T22:03:10.1004651Z ValueError: Error creating optimizer.
2023-01-11T22:03:10.1004658Z 
2023-01-11T22:03:10.1005544Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.OptimizerFailingOnConstructor'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2023-01-11T22:03:10.1005642Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.1005783Z ValueError('Error creating optimizer.')
2023-01-11T22:03:10.1005875Z Traceback (most recent call last):
2023-01-11T22:03:10.1006135Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1006269Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1006546Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 86, in _new_local_optimizer
2023-01-11T22:03:10.1006688Z     return rpc.RRef(_LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2023-01-11T22:03:10.1006948Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 74, in __init__
2023-01-11T22:03:10.1007073Z     self.optim = optim_cls(self._local_params, *args, **kwargs)
2023-01-11T22:03:10.1007374Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2023-01-11T22:03:10.1007488Z     raise ValueError("Error creating optimizer.")
2023-01-11T22:03:10.1007588Z ValueError: Error creating optimizer.
2023-01-11T22:03:10.1007594Z 
2023-01-11T22:03:10.1008423Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.OptimizerFailingOnConstructor'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2023-01-11T22:03:10.1009238Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.OptimizerFailingOnConstructor'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2023-01-11T22:03:10.1009333Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.1009474Z ValueError('Error creating optimizer.')
2023-01-11T22:03:10.1009566Z Traceback (most recent call last):
2023-01-11T22:03:10.1009815Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1009951Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1010230Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 86, in _new_local_optimizer
2023-01-11T22:03:10.1010385Z     return rpc.RRef(_LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2023-01-11T22:03:10.1010643Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 74, in __init__
2023-01-11T22:03:10.1010768Z     self.optim = optim_cls(self._local_params, *args, **kwargs)
2023-01-11T22:03:10.1011065Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2023-01-11T22:03:10.1011179Z     raise ValueError("Error creating optimizer.")
2023-01-11T22:03:10.1011267Z ValueError: Error creating optimizer.
2023-01-11T22:03:10.1011272Z 
2023-01-11T22:03:10.1011367Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.1011506Z ValueError('Error creating optimizer.')
2023-01-11T22:03:10.1011599Z Traceback (most recent call last):
2023-01-11T22:03:10.1011889Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1012051Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1012328Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 86, in _new_local_optimizer
2023-01-11T22:03:10.1012480Z     return rpc.RRef(_LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2023-01-11T22:03:10.1012723Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 74, in __init__
2023-01-11T22:03:10.1012847Z     self.optim = optim_cls(self._local_params, *args, **kwargs)
2023-01-11T22:03:10.1013146Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2023-01-11T22:03:10.1013262Z     raise ValueError("Error creating optimizer.")
2023-01-11T22:03:10.1013363Z ValueError: Error creating optimizer.
2023-01-11T22:03:10.1013370Z 
2023-01-11T22:03:10.1013463Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.1013601Z ValueError('Error creating optimizer.')
2023-01-11T22:03:10.1013695Z Traceback (most recent call last):
2023-01-11T22:03:10.1013940Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1014073Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1014346Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 86, in _new_local_optimizer
2023-01-11T22:03:10.1014498Z     return rpc.RRef(_LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2023-01-11T22:03:10.1014755Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 74, in __init__
2023-01-11T22:03:10.1014879Z     self.optim = optim_cls(self._local_params, *args, **kwargs)
2023-01-11T22:03:10.1015172Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2023-01-11T22:03:10.1015289Z     raise ValueError("Error creating optimizer.")
2023-01-11T22:03:10.1015376Z ValueError: Error creating optimizer.
2023-01-11T22:03:10.1015395Z 
2023-01-11T22:03:10.1015474Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.1015613Z ValueError('Error creating optimizer.')
2023-01-11T22:03:10.1015705Z Traceback (most recent call last):
2023-01-11T22:03:10.1015964Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1016098Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1016374Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 86, in _new_local_optimizer
2023-01-11T22:03:10.1016525Z     return rpc.RRef(_LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2023-01-11T22:03:10.1016783Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 74, in __init__
2023-01-11T22:03:10.1016895Z     self.optim = optim_cls(self._local_params, *args, **kwargs)
2023-01-11T22:03:10.1017190Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2023-01-11T22:03:10.1017302Z     raise ValueError("Error creating optimizer.")
2023-01-11T22:03:10.1017404Z ValueError: Error creating optimizer.
2023-01-11T22:03:10.1017410Z 
2023-01-11T22:03:10.1017501Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.1017639Z ValueError('Error creating optimizer.')
2023-01-11T22:03:10.1017731Z Traceback (most recent call last):
2023-01-11T22:03:10.1017972Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1018106Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1018380Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 86, in _new_local_optimizer
2023-01-11T22:03:10.1018565Z     return rpc.RRef(_LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2023-01-11T22:03:10.1018849Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 74, in __init__
2023-01-11T22:03:10.1018974Z     self.optim = optim_cls(self._local_params, *args, **kwargs)
2023-01-11T22:03:10.1019268Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2023-01-11T22:03:10.1019382Z     raise ValueError("Error creating optimizer.")
2023-01-11T22:03:10.1019482Z ValueError: Error creating optimizer.
2023-01-11T22:03:10.1019487Z 
2023-01-11T22:03:10.1019566Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.1019703Z ValueError('Error creating optimizer.')
2023-01-11T22:03:10.1019795Z Traceback (most recent call last):
2023-01-11T22:03:10.1020050Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1020188Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1020458Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 86, in _new_local_optimizer
2023-01-11T22:03:10.1020609Z     return rpc.RRef(_LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2023-01-11T22:03:10.1020863Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 74, in __init__
2023-01-11T22:03:10.1020972Z     self.optim = optim_cls(self._local_params, *args, **kwargs)
2023-01-11T22:03:10.1021264Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2023-01-11T22:03:10.1021377Z     raise ValueError("Error creating optimizer.")
2023-01-11T22:03:10.1021476Z ValueError: Error creating optimizer.
2023-01-11T22:03:10.1021482Z 
2023-01-11T22:03:10.1021550Z ok (2.742s)
2023-01-11T22:03:10.1021557Z 
2023-01-11T22:03:10.1021942Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1022053Z Ran 1 test in 2.742s
2023-01-11T22:03:10.1022058Z 
2023-01-11T22:03:10.1022123Z OK
2023-01-11T22:03:10.1022128Z 
2023-01-11T22:03:10.1022201Z Generating XML reports...
2023-01-11T22:03:10.1022590Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20230111213416.xml
2023-01-11T22:03:10.1022872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1022999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1023286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1023425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1023614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsot6bv1t
2023-01-11T22:03:10.1023819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsot6bv1t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1024080Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1024086Z 
2023-01-11T22:03:10.1024146Z Running tests...
2023-01-11T22:03:10.1024346Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1025048Z   test_dist_optim_none_grads (__main__.TensorPipeDistOptimizerTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77075 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.318s)
2023-01-11T22:03:10.1025054Z 
2023-01-11T22:03:10.1025250Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1025403Z Ran 1 test in 0.319s
2023-01-11T22:03:10.1025409Z 
2023-01-11T22:03:10.1025482Z OK (skipped=1)
2023-01-11T22:03:10.1025487Z 
2023-01-11T22:03:10.1025626Z Generating XML reports...
2023-01-11T22:03:10.1026010Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20230111213421.xml
2023-01-11T22:03:10.1026288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1026415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1026688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1026828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1027017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn6kkjax2
2023-01-11T22:03:10.1027217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn6kkjax2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1027474Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1027479Z 
2023-01-11T22:03:10.1027555Z Running tests...
2023-01-11T22:03:10.1027752Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1027995Z   test_dist_backward (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20286
2023-01-11T22:03:10.1028153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20287
2023-01-11T22:03:10.1028295Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20288
2023-01-11T22:03:10.1028452Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20289
2023-01-11T22:03:10.1028730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1028859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1029147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1029285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1029474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakai1raf
2023-01-11T22:03:10.1029673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakai1raf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1029934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1030061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1030341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1030482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1030670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_gsy6p3
2023-01-11T22:03:10.1030867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_gsy6p3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1031139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1031263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1031543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1031666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1031850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6o63fql
2023-01-11T22:03:10.1032043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6o63fql/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1032378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1032504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1032785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1032921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1033106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9c2yl80m
2023-01-11T22:03:10.1033289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9c2yl80m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1033454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1033621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1033788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1033954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1034023Z ok (2.654s)
2023-01-11T22:03:10.1034029Z 
2023-01-11T22:03:10.1034229Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1034306Z Ran 1 test in 2.654s
2023-01-11T22:03:10.1034312Z 
2023-01-11T22:03:10.1034373Z OK
2023-01-11T22:03:10.1034379Z 
2023-01-11T22:03:10.1034451Z Generating XML reports...
2023-01-11T22:03:10.1034831Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20230111213423.xml
2023-01-11T22:03:10.1035108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1035234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1035519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1035659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1035846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpblljswht
2023-01-11T22:03:10.1036043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpblljswht/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1036286Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1036304Z 
2023-01-11T22:03:10.1036365Z Running tests...
2023-01-11T22:03:10.1036561Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1036802Z   test_get_gradients (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20519
2023-01-11T22:03:10.1036962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20520
2023-01-11T22:03:10.1037118Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20521
2023-01-11T22:03:10.1037276Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20522
2023-01-11T22:03:10.1037552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1037676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1037946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1038082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1038268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2e8d56q
2023-01-11T22:03:10.1038466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2e8d56q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1038774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1039015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1039307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1039443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1039631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4hwm8r1
2023-01-11T22:03:10.1039812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4hwm8r1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1040085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1040210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1040499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1040639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1040826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp281gc8e4
2023-01-11T22:03:10.1041021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp281gc8e4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1041292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1041401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1041681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1041816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1042000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2vdsvg8
2023-01-11T22:03:10.1042196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2vdsvg8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1042362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1042526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1042690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1042852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1042907Z ok (2.742s)
2023-01-11T22:03:10.1042913Z 
2023-01-11T22:03:10.1043111Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1043188Z Ran 1 test in 2.742s
2023-01-11T22:03:10.1043193Z 
2023-01-11T22:03:10.1043254Z OK
2023-01-11T22:03:10.1043259Z 
2023-01-11T22:03:10.1043345Z Generating XML reports...
2023-01-11T22:03:10.1043728Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20230111213428.xml
2023-01-11T22:03:10.1044006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1044132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1044399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1044535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1044722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61iglgys
2023-01-11T22:03:10.1044918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61iglgys/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1045176Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1045217Z 
2023-01-11T22:03:10.1045291Z Running tests...
2023-01-11T22:03:10.1045488Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1045765Z   test_jit_fork_within_context (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20762
2023-01-11T22:03:10.1045924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20763
2023-01-11T22:03:10.1046068Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20764
2023-01-11T22:03:10.1046221Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20765
2023-01-11T22:03:10.1046497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1046623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1046908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1047048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1047238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5syko43
2023-01-11T22:03:10.1047434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5syko43/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1047698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1047822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1048107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1048243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1048429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmexqtz_d
2023-01-11T22:03:10.1048703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1048830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1049109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1049245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1049417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehtfjngl
2023-01-11T22:03:10.1049689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1049813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1050095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1050233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1050420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpymef25il
2023-01-11T22:03:10.1050627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmexqtz_d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1050823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehtfjngl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1051010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpymef25il/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1051163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1051327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1051491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1051653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1051754Z ok (2.753s)
2023-01-11T22:03:10.1051759Z 
2023-01-11T22:03:10.1051988Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1052066Z Ran 1 test in 2.753s
2023-01-11T22:03:10.1052071Z 
2023-01-11T22:03:10.1052132Z OK
2023-01-11T22:03:10.1052136Z 
2023-01-11T22:03:10.1052209Z Generating XML reports...
2023-01-11T22:03:10.1052590Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20230111213432.xml
2023-01-11T22:03:10.1052866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1052994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1053275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1053411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1053600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpixiu7hvm
2023-01-11T22:03:10.1053800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpixiu7hvm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1054056Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1054061Z 
2023-01-11T22:03:10.1054122Z Running tests...
2023-01-11T22:03:10.1054319Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1054588Z   test_restore_context_after_swtich_to_jit_thread (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21005
2023-01-11T22:03:10.1054747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21006
2023-01-11T22:03:10.1054904Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21007
2023-01-11T22:03:10.1055061Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21008
2023-01-11T22:03:10.1055340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1055465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1055734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1055871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1056056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5gm9gju
2023-01-11T22:03:10.1056254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5gm9gju/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1056527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1056652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1056934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1057070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1057256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu96pinz3
2023-01-11T22:03:10.1057436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu96pinz3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1057708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1057833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1058111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1058248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1058468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8lgcp5sa
2023-01-11T22:03:10.1058692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8lgcp5sa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1058968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1059079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1059356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1059495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1059680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgf0fwflq
2023-01-11T22:03:10.1059877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgf0fwflq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1060045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1060213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1060377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1060541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1060596Z ok (2.840s)
2023-01-11T22:03:10.1060601Z 
2023-01-11T22:03:10.1060803Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1060878Z Ran 1 test in 2.840s
2023-01-11T22:03:10.1060884Z 
2023-01-11T22:03:10.1060945Z OK
2023-01-11T22:03:10.1060950Z 
2023-01-11T22:03:10.1061037Z Generating XML reports...
2023-01-11T22:03:10.1061418Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20230111213437.xml
2023-01-11T22:03:10.1061827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1061957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1062242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1062365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1062552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphvo536wi
2023-01-11T22:03:10.1062748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphvo536wi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1063005Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1063010Z 
2023-01-11T22:03:10.1063086Z Running tests...
2023-01-11T22:03:10.1063283Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1063517Z   test_add_done_callback (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21238
2023-01-11T22:03:10.1063679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21239
2023-01-11T22:03:10.1063821Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21240
2023-01-11T22:03:10.1063979Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21241
2023-01-11T22:03:10.1064255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1064382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1064669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1064805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1065120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkeff1lm7
2023-01-11T22:03:10.1065405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkeff1lm7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1065689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1065802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1066084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1066221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1066407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfd3jvurk
2023-01-11T22:03:10.1066605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfd3jvurk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1066878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1067006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1067289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1067410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1067596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhzspu7i
2023-01-11T22:03:10.1067792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhzspu7i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1068067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1068192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1068476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1068614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1068801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptmvcjrf9
2023-01-11T22:03:10.1068996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptmvcjrf9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1069149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1069315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1069480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1069644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1069713Z ok (2.751s)
2023-01-11T22:03:10.1069719Z 
2023-01-11T22:03:10.1069916Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1069994Z Ran 1 test in 2.751s
2023-01-11T22:03:10.1069999Z 
2023-01-11T22:03:10.1070060Z OK
2023-01-11T22:03:10.1070066Z 
2023-01-11T22:03:10.1070139Z Generating XML reports...
2023-01-11T22:03:10.1070491Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213442.xml
2023-01-11T22:03:10.1070765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1070889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1071172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1071310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1071494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytboe_s6
2023-01-11T22:03:10.1071719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytboe_s6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1072015Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1072021Z 
2023-01-11T22:03:10.1072083Z Running tests...
2023-01-11T22:03:10.1072281Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1072529Z   test_all_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21477
2023-01-11T22:03:10.1072689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21478
2023-01-11T22:03:10.1072845Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21479
2023-01-11T22:03:10.1073000Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21480
2023-01-11T22:03:10.1073277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1073404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1073683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1073885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1074076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5p1k1rb
2023-01-11T22:03:10.1074271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5p1k1rb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1074546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1074672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1074954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1075093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1075280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgee4t84
2023-01-11T22:03:10.1075463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgee4t84/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1075788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1075963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1076284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1076420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1076605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wace8xr
2023-01-11T22:03:10.1076800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wace8xr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1077076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1077202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1077472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1077608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1077793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfr_cvzz
2023-01-11T22:03:10.1077988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfr_cvzz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1078153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1078317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1078522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1078715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1078772Z ok (2.846s)
2023-01-11T22:03:10.1078777Z 
2023-01-11T22:03:10.1079049Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1079126Z Ran 1 test in 2.846s
2023-01-11T22:03:10.1079132Z 
2023-01-11T22:03:10.1079194Z OK
2023-01-11T22:03:10.1079199Z 
2023-01-11T22:03:10.1079286Z Generating XML reports...
2023-01-11T22:03:10.1079641Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213446.xml
2023-01-11T22:03:10.1079921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1080048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1080335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1080460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1080647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6cm42tzu
2023-01-11T22:03:10.1080845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6cm42tzu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1081104Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1081109Z 
2023-01-11T22:03:10.1081187Z Running tests...
2023-01-11T22:03:10.1081390Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1081645Z   test_args_and_kwargs_contain_different_types (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21704
2023-01-11T22:03:10.1081809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21705
2023-01-11T22:03:10.1081951Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21706
2023-01-11T22:03:10.1082108Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21707
2023-01-11T22:03:10.1082384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1082509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1082793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1082929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1083118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdk5o34o
2023-01-11T22:03:10.1083315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdk5o34o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1083591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1083703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1083985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1084120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1084306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphgqa7ns0
2023-01-11T22:03:10.1084501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphgqa7ns0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1084774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1084898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1085216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1085376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1085563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7oksz4_t
2023-01-11T22:03:10.1085759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7oksz4_t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1086034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1086159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1086438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1086573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1086758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26mzshd5
2023-01-11T22:03:10.1086959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26mzshd5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1087112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1087279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1087442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1087604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1087672Z ok (2.862s)
2023-01-11T22:03:10.1087677Z 
2023-01-11T22:03:10.1087878Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1087954Z Ran 1 test in 2.862s
2023-01-11T22:03:10.1087960Z 
2023-01-11T22:03:10.1088021Z OK
2023-01-11T22:03:10.1088028Z 
2023-01-11T22:03:10.1088099Z Generating XML reports...
2023-01-11T22:03:10.1088455Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213451.xml
2023-01-11T22:03:10.1088732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1088858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1089140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1089276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1089461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp440xt2ze
2023-01-11T22:03:10.1089656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp440xt2ze/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1089913Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1089921Z 
2023-01-11T22:03:10.1089982Z Running tests...
2023-01-11T22:03:10.1090180Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1090427Z   test_args_kwargs_are_neither_passed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21931
2023-01-11T22:03:10.1090588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21932
2023-01-11T22:03:10.1090742Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21933
2023-01-11T22:03:10.1090896Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21934
2023-01-11T22:03:10.1091171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1091295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1091578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1091746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1091961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsmfu1fhi
2023-01-11T22:03:10.1092158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsmfu1fhi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1092434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1092558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1092843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1092978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1093162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw79bck8s
2023-01-11T22:03:10.1093346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw79bck8s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1093618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1093742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1094025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1094160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1094346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2wkaoj4
2023-01-11T22:03:10.1094542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2wkaoj4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1094814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1094940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1095208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1095344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1095531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu29ol5op
2023-01-11T22:03:10.1095727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu29ol5op/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1095893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1096058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1096225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1096386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1096443Z ok (2.750s)
2023-01-11T22:03:10.1096466Z 
2023-01-11T22:03:10.1096654Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1096730Z Ran 1 test in 2.750s
2023-01-11T22:03:10.1096736Z 
2023-01-11T22:03:10.1096799Z OK
2023-01-11T22:03:10.1096804Z 
2023-01-11T22:03:10.1096889Z Generating XML reports...
2023-01-11T22:03:10.1097242Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213456.xml
2023-01-11T22:03:10.1097515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1097638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1097920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1098043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1098258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8k9jql7l
2023-01-11T22:03:10.1098480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8k9jql7l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1098740Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1098745Z 
2023-01-11T22:03:10.1098819Z Running tests...
2023-01-11T22:03:10.1099016Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1099247Z   test_async_function_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22158
2023-01-11T22:03:10.1099406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22159
2023-01-11T22:03:10.1099548Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22160
2023-01-11T22:03:10.1099705Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22161
2023-01-11T22:03:10.1099981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1100108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1100398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1100535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1100721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ptu3bto
2023-01-11T22:03:10.1100916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ptu3bto/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1101193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1101304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1101591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1101857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1102047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpga17nz8v
2023-01-11T22:03:10.1102245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpga17nz8v/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1102524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1102650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1102932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1103068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1103242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_iiuknr
2023-01-11T22:03:10.1103522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1103647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1103929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1104066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1104254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3nv4h56c
2023-01-11T22:03:10.1104451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_iiuknr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1104630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3nv4h56c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1104801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1105062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1105260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1105426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1105496Z ok (2.752s)
2023-01-11T22:03:10.1105501Z 
2023-01-11T22:03:10.1105702Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1105777Z Ran 1 test in 2.752s
2023-01-11T22:03:10.1105782Z 
2023-01-11T22:03:10.1105830Z OK
2023-01-11T22:03:10.1105848Z 
2023-01-11T22:03:10.1105920Z Generating XML reports...
2023-01-11T22:03:10.1106273Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213501.xml
2023-01-11T22:03:10.1106549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1106678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1106966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1107103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1107289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9jn3tvw
2023-01-11T22:03:10.1107484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9jn3tvw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1107726Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1107732Z 
2023-01-11T22:03:10.1107808Z Running tests...
2023-01-11T22:03:10.1108005Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1108245Z   test_async_function_remote_multi (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22381
2023-01-11T22:03:10.1108410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22382
2023-01-11T22:03:10.1108567Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22383
2023-01-11T22:03:10.1108725Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22384
2023-01-11T22:03:10.1109003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1109114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1109398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1109535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1109722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdt6o0a6z
2023-01-11T22:03:10.1109922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdt6o0a6z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1110200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1110326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1110609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1110745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1110919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ujm1zju
2023-01-11T22:03:10.1111116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ujm1zju/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1111389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1111544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1111854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1111992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1112178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxlnrj_q3
2023-01-11T22:03:10.1112374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxlnrj_q3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1112648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1112760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1113042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1113180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1113369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe531y27x
2023-01-11T22:03:10.1113564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe531y27x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1113731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1113897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1114061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1114210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1114279Z ok (2.839s)
2023-01-11T22:03:10.1114285Z 
2023-01-11T22:03:10.1114484Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1114563Z Ran 1 test in 2.840s
2023-01-11T22:03:10.1114568Z 
2023-01-11T22:03:10.1114630Z OK
2023-01-11T22:03:10.1114636Z 
2023-01-11T22:03:10.1114724Z Generating XML reports...
2023-01-11T22:03:10.1115079Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213505.xml
2023-01-11T22:03:10.1115355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1115479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1115748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1115885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1116072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzww9tnkw
2023-01-11T22:03:10.1116268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzww9tnkw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1116522Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1116527Z 
2023-01-11T22:03:10.1116601Z Running tests...
2023-01-11T22:03:10.1116798Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1117025Z   test_async_function_simple (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22604
2023-01-11T22:03:10.1117172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22605
2023-01-11T22:03:10.1117328Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22606
2023-01-11T22:03:10.1117483Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22607
2023-01-11T22:03:10.1117759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1117912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1118238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1118373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1118559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7j2qwki
2023-01-11T22:03:10.1118753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7j2qwki/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1119109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1119235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1119517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1119653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1119841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0fby7oc
2023-01-11T22:03:10.1120038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0fby7oc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1120311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1120434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1120701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1120835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1121020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhb_2rly
2023-01-11T22:03:10.1121216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhb_2rly/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1121491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1121618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1121899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1122032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1122216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_0fx_pc
2023-01-11T22:03:10.1122394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_0fx_pc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1122560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1122724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1122887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1123051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1123121Z ok (2.742s)
2023-01-11T22:03:10.1123127Z 
2023-01-11T22:03:10.1123325Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1123398Z Ran 1 test in 2.743s
2023-01-11T22:03:10.1123404Z 
2023-01-11T22:03:10.1123452Z OK
2023-01-11T22:03:10.1123469Z 
2023-01-11T22:03:10.1123541Z Generating XML reports...
2023-01-11T22:03:10.1123888Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213510.xml
2023-01-11T22:03:10.1124163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1124287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1124566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1124737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1124950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkus_luei
2023-01-11T22:03:10.1125145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkus_luei/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1125390Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1125396Z 
2023-01-11T22:03:10.1125467Z Running tests...
2023-01-11T22:03:10.1125664Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1125913Z   test_async_function_wrong_decorator_order (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22827
2023-01-11T22:03:10.1126069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22828
2023-01-11T22:03:10.1126227Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22829
2023-01-11T22:03:10.1126385Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22830
2023-01-11T22:03:10.1126661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1126786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1127056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1127191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1127377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptaeww_r2
2023-01-11T22:03:10.1127571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptaeww_r2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1127844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1127970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1128253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1128388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1128558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ogbhodz
2023-01-11T22:03:10.1128751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ogbhodz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1129023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1129146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1129428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1129564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1129752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivcrw8p8
2023-01-11T22:03:10.1129949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivcrw8p8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1130219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1130330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1130610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1130745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1130929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph88k1mnh
2023-01-11T22:03:10.1131152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph88k1mnh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1131345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1131510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1131672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1131821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1131887Z ok (2.637s)
2023-01-11T22:03:10.1131892Z 
2023-01-11T22:03:10.1132089Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1132163Z Ran 1 test in 2.637s
2023-01-11T22:03:10.1132169Z 
2023-01-11T22:03:10.1132227Z OK
2023-01-11T22:03:10.1132232Z 
2023-01-11T22:03:10.1132317Z Generating XML reports...
2023-01-11T22:03:10.1132669Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213515.xml
2023-01-11T22:03:10.1132948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1133070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1133338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1133472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1133656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_9_26hv
2023-01-11T22:03:10.1133845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_9_26hv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1134098Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1134103Z 
2023-01-11T22:03:10.1134179Z Running tests...
2023-01-11T22:03:10.1134378Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1134622Z   test_async_function_wrong_return_type (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23050
2023-01-11T22:03:10.1134768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23051
2023-01-11T22:03:10.1134922Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23052
2023-01-11T22:03:10.1135075Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23053
2023-01-11T22:03:10.1135354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1135476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1135756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1135894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1136084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpefhe7d5w
2023-01-11T22:03:10.1136279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpefhe7d5w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1136538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1136660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1136939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1137073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1137256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ftse_c8
2023-01-11T22:03:10.1137449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ftse_c8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1137751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1137901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1138171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1138304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1138488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpex3crni3
2023-01-11T22:03:10.1138758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1138879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1139156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1139293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1139481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn09810o5
2023-01-11T22:03:10.1139675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpex3crni3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1139850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn09810o5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1140014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1140176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1140341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1140501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1140571Z ok (2.650s)
2023-01-11T22:03:10.1140576Z 
2023-01-11T22:03:10.1140771Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1140849Z Ran 1 test in 2.650s
2023-01-11T22:03:10.1140857Z 
2023-01-11T22:03:10.1140906Z OK
2023-01-11T22:03:10.1140922Z 
2023-01-11T22:03:10.1140993Z Generating XML reports...
2023-01-11T22:03:10.1141344Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213519.xml
2023-01-11T22:03:10.1141619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1141866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1142149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1142286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1142478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4js4jtfq
2023-01-11T22:03:10.1142676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4js4jtfq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1142918Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1142939Z 
2023-01-11T22:03:10.1143000Z Running tests...
2023-01-11T22:03:10.1143200Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1143453Z   test_async_function_wrong_return_type_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23273
2023-01-11T22:03:10.1143613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23274
2023-01-11T22:03:10.1143771Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23275
2023-01-11T22:03:10.1143929Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23276
2023-01-11T22:03:10.1144256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1144418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1144689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1144826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1145013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbcrs1cz
2023-01-11T22:03:10.1145287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1145411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1145693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1145833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1146019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpolvbqk11
2023-01-11T22:03:10.1146205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbcrs1cz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1146397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpolvbqk11/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1146670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1146795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1147072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1147208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1147391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfauzewbt
2023-01-11T22:03:10.1147587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfauzewbt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1147860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1147971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1148247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1148382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1148563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2pu7o9gs
2023-01-11T22:03:10.1148755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2pu7o9gs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1148919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1149086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1149250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1149398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1149464Z ok (2.661s)
2023-01-11T22:03:10.1149471Z 
2023-01-11T22:03:10.1149666Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1149741Z Ran 1 test in 2.662s
2023-01-11T22:03:10.1149746Z 
2023-01-11T22:03:10.1149806Z OK
2023-01-11T22:03:10.1149812Z 
2023-01-11T22:03:10.1149894Z Generating XML reports...
2023-01-11T22:03:10.1150245Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213524.xml
2023-01-11T22:03:10.1150518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1150675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1150976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1151111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1151296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5xam573
2023-01-11T22:03:10.1151491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5xam573/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1151744Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1151750Z 
2023-01-11T22:03:10.1151822Z Running tests...
2023-01-11T22:03:10.1152022Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1152247Z   test_async_script_throw (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23496
2023-01-11T22:03:10.1152406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23497
2023-01-11T22:03:10.1152551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23498
2023-01-11T22:03:10.1152703Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23499
2023-01-11T22:03:10.1152977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1153099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1153380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1153514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1153698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20q0puwp
2023-01-11T22:03:10.1153896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20q0puwp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1154159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1154281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1154561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1154696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1154878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zxtk_wq
2023-01-11T22:03:10.1155069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zxtk_wq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1155341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1155465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1155749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1155870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1156052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfiiivu8l
2023-01-11T22:03:10.1156323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1156445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1156723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1156856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1157040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu97f7s9u
2023-01-11T22:03:10.1157264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfiiivu8l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1157466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu97f7s9u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1157633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1157795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1157956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1158120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1158188Z ok (2.747s)
2023-01-11T22:03:10.1158193Z 
2023-01-11T22:03:10.1158392Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1158470Z Ran 1 test in 2.747s
2023-01-11T22:03:10.1158478Z 
2023-01-11T22:03:10.1158525Z OK
2023-01-11T22:03:10.1158542Z 
2023-01-11T22:03:10.1158612Z Generating XML reports...
2023-01-11T22:03:10.1159042Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213528.xml
2023-01-11T22:03:10.1159324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1159450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1159733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1159869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1160055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuypb2rt5
2023-01-11T22:03:10.1160253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuypb2rt5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1160497Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1160515Z 
2023-01-11T22:03:10.1160578Z Running tests...
2023-01-11T22:03:10.1160777Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1161006Z   test_async_script_udf (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23735
2023-01-11T22:03:10.1161165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23736
2023-01-11T22:03:10.1161323Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23737
2023-01-11T22:03:10.1161478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23738
2023-01-11T22:03:10.1161757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1161881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1162154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1162293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1162479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgust0mc9
2023-01-11T22:03:10.1162675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgust0mc9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1162950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1163076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1163359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1163495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1163699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcqowivs
2023-01-11T22:03:10.1163936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcqowivs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1164209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1164331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1164611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1164746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1164932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgk_0y7k
2023-01-11T22:03:10.1165203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1165328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1165597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1165736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1165923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2a7j3xc
2023-01-11T22:03:10.1166118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgk_0y7k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1166309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2a7j3xc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1166478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1166644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1166809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1166961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1167033Z ok (2.743s)
2023-01-11T22:03:10.1167039Z 
2023-01-11T22:03:10.1167238Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1167315Z Ran 1 test in 2.744s
2023-01-11T22:03:10.1167320Z 
2023-01-11T22:03:10.1167382Z OK
2023-01-11T22:03:10.1167387Z 
2023-01-11T22:03:10.1167472Z Generating XML reports...
2023-01-11T22:03:10.1167824Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213533.xml
2023-01-11T22:03:10.1168099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1168225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1168494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1168632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1168820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4v7dt6qw
2023-01-11T22:03:10.1169016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4v7dt6qw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1169275Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1169281Z 
2023-01-11T22:03:10.1169353Z Running tests...
2023-01-11T22:03:10.1169548Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1169791Z   test_call_fork_in_jit_with_profiling (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23974
2023-01-11T22:03:10.1169951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23975
2023-01-11T22:03:10.1170095Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23976
2023-01-11T22:03:10.1170284Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23977
2023-01-11T22:03:10.1170583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1170710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1170989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1171123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1171308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1889oa8o
2023-01-11T22:03:10.1171502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1889oa8o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1171761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1171884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1172166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1172299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1172481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplv8zyt6x
2023-01-11T22:03:10.1172674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplv8zyt6x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1172945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1173065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1173347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1173470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1173652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3s99stoy
2023-01-11T22:03:10.1173846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3s99stoy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1174117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1174238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1174614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1174755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1174938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsa6mwpup
2023-01-11T22:03:10.1175122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsa6mwpup/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1175287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1175450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1175611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1175771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1175838Z ok (2.344s)
2023-01-11T22:03:10.1175843Z 
2023-01-11T22:03:10.1176043Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1176117Z Ran 1 test in 2.344s
2023-01-11T22:03:10.1176123Z 
2023-01-11T22:03:10.1176182Z OK
2023-01-11T22:03:10.1176187Z 
2023-01-11T22:03:10.1176258Z Generating XML reports...
2023-01-11T22:03:10.1176607Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213538.xml
2023-01-11T22:03:10.1176918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1177070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1177351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1177485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1177669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuokmov3b
2023-01-11T22:03:10.1177864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuokmov3b/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1178106Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1178123Z 
2023-01-11T22:03:10.1178184Z Running tests...
2023-01-11T22:03:10.1178378Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1178647Z   test_call_python_function_remotely_from_script_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24045
2023-01-11T22:03:10.1178807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24046
2023-01-11T22:03:10.1178959Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24047
2023-01-11T22:03:10.1179113Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24048
2023-01-11T22:03:10.1179386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1179507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1179777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1179909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1180096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvexoll5k
2023-01-11T22:03:10.1180294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvexoll5k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1180567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1180692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1180974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1181111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1181284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnut9pyqy
2023-01-11T22:03:10.1181478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnut9pyqy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1181865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1181991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1182272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1182405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1182589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpztx5x7nf
2023-01-11T22:03:10.1182784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpztx5x7nf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1183054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1183164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1183500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1183671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1183857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jzw7mv9
2023-01-11T22:03:10.1184051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jzw7mv9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1184216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1184380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1184543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1184706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1184761Z ok (2.743s)
2023-01-11T22:03:10.1184769Z 
2023-01-11T22:03:10.1184968Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1185044Z Ran 1 test in 2.743s
2023-01-11T22:03:10.1185050Z 
2023-01-11T22:03:10.1185113Z OK
2023-01-11T22:03:10.1185120Z 
2023-01-11T22:03:10.1185205Z Generating XML reports...
2023-01-11T22:03:10.1185555Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213542.xml
2023-01-11T22:03:10.1185830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1185953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1186224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1186359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1186543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvcu7jhp
2023-01-11T22:03:10.1186741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvcu7jhp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1187000Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1187005Z 
2023-01-11T22:03:10.1187076Z Running tests...
2023-01-11T22:03:10.1187271Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1187507Z   test_call_rpc_with_profiling (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24272
2023-01-11T22:03:10.1187663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24273
2023-01-11T22:03:10.1187806Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24274
2023-01-11T22:03:10.1187958Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24275
2023-01-11T22:03:10.1188235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1188358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1188640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1188778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1188962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpijaev1fv
2023-01-11T22:03:10.1189156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpijaev1fv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1189416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1189538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1189816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1189983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1190203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8sn5a3r
2023-01-11T22:03:10.1190475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1190598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1190876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1191008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1191179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qvra2ht
2023-01-11T22:03:10.1191375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8sn5a3r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1191566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qvra2ht/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1191841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1191963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1192238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1192371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1192553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4d8bfxtt
2023-01-11T22:03:10.1192734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4d8bfxtt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1192900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1193065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1193229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1193393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1193459Z ok (2.744s)
2023-01-11T22:03:10.1193465Z 
2023-01-11T22:03:10.1193661Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1193734Z Ran 1 test in 2.745s
2023-01-11T22:03:10.1193739Z 
2023-01-11T22:03:10.1193799Z OK
2023-01-11T22:03:10.1193804Z 
2023-01-11T22:03:10.1193875Z Generating XML reports...
2023-01-11T22:03:10.1194223Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213547.xml
2023-01-11T22:03:10.1194495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1194624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1194909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1195044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1195228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdkc_8cn_
2023-01-11T22:03:10.1195424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdkc_8cn_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1195670Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1195686Z 
2023-01-11T22:03:10.1195746Z Running tests...
2023-01-11T22:03:10.1195943Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1196209Z   test_call_script_function_that_not_exists_remotely_from_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24499
2023-01-11T22:03:10.1196407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24500
2023-01-11T22:03:10.1196589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24501
2023-01-11T22:03:10.1196743Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24502
2023-01-11T22:03:10.1197020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1197144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1197414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1197548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1197732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1p8yvk6v
2023-01-11T22:03:10.1198010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1198133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1198417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1198552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1198736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchechlhc
2023-01-11T22:03:10.1199006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1p8yvk6v/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1199192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchechlhc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1199467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1199596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1199879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1200018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1200201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkb_swji7
2023-01-11T22:03:10.1200398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkb_swji7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1200672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1200782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1201064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1201199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1201386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkbacdv9m
2023-01-11T22:03:10.1201584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkbacdv9m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1201748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1201911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1202073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1202236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1202290Z ok (2.750s)
2023-01-11T22:03:10.1202296Z 
2023-01-11T22:03:10.1202493Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1202567Z Ran 1 test in 2.750s
2023-01-11T22:03:10.1202572Z 
2023-01-11T22:03:10.1202666Z OK
2023-01-11T22:03:10.1202671Z 
2023-01-11T22:03:10.1202754Z Generating XML reports...
2023-01-11T22:03:10.1203169Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213551.xml
2023-01-11T22:03:10.1203444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1203570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1203838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1203973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1204157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76wxcq94
2023-01-11T22:03:10.1204350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76wxcq94/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1204609Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1204614Z 
2023-01-11T22:03:10.1204688Z Running tests...
2023-01-11T22:03:10.1204889Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1205152Z   test_call_script_function_that_raises_remotely_from_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24726
2023-01-11T22:03:10.1205312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24727
2023-01-11T22:03:10.1205455Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24728
2023-01-11T22:03:10.1205608Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24729
2023-01-11T22:03:10.1205887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1206012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1206298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1206437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1206625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkwy9ly9
2023-01-11T22:03:10.1206827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkwy9ly9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1207102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1207212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1207495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1207630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1207818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplg9zqo_7
2023-01-11T22:03:10.1208016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplg9zqo_7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1208288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1208411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1208693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1208815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1208998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvoxtyx7
2023-01-11T22:03:10.1209190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvoxtyx7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1209463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1209621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1209941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1210076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1210262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy07z1nn_
2023-01-11T22:03:10.1210454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy07z1nn_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1210606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1210770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1210935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1211098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1211164Z ok (2.745s)
2023-01-11T22:03:10.1211170Z 
2023-01-11T22:03:10.1211370Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1211445Z Ran 1 test in 2.745s
2023-01-11T22:03:10.1211450Z 
2023-01-11T22:03:10.1211510Z OK
2023-01-11T22:03:10.1211515Z 
2023-01-11T22:03:10.1211586Z Generating XML reports...
2023-01-11T22:03:10.1211937Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213556.xml
2023-01-11T22:03:10.1212213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1212339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1212622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1212761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1212952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgws9pxea
2023-01-11T22:03:10.1213148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgws9pxea/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1213404Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1213410Z 
2023-01-11T22:03:10.1213472Z Running tests...
2023-01-11T22:03:10.1213668Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1213892Z   test_callback_chain (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24953
2023-01-11T22:03:10.1214052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24954
2023-01-11T22:03:10.1214208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24955
2023-01-11T22:03:10.1214364Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24956
2023-01-11T22:03:10.1214646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1214772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1215040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1215178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1215363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1sqhvm0
2023-01-11T22:03:10.1215557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1sqhvm0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1215831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1215993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1216309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1216448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1216637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmttsrb5i
2023-01-11T22:03:10.1216822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmttsrb5i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1217097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1217222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1217510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1217647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1217835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmpcgbjw4
2023-01-11T22:03:10.1218034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmpcgbjw4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1218308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1218420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1218702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1218838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1219022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdrfstf9i
2023-01-11T22:03:10.1219219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdrfstf9i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1219387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1219557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1219723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1219887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1219943Z ok (2.747s)
2023-01-11T22:03:10.1219950Z 
2023-01-11T22:03:10.1220148Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1220224Z Ran 1 test in 2.747s
2023-01-11T22:03:10.1220230Z 
2023-01-11T22:03:10.1220292Z OK
2023-01-11T22:03:10.1220297Z 
2023-01-11T22:03:10.1220382Z Generating XML reports...
2023-01-11T22:03:10.1220731Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213601.xml
2023-01-11T22:03:10.1221008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1221135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1221406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1221543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1221837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gyyx49a
2023-01-11T22:03:10.1222034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gyyx49a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1222292Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1222297Z 
2023-01-11T22:03:10.1222374Z Running tests...
2023-01-11T22:03:10.1222573Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1222889Z   test_callback_simple (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25176
2023-01-11T22:03:10.1223051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25177
2023-01-11T22:03:10.1223193Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25178
2023-01-11T22:03:10.1223348Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25179
2023-01-11T22:03:10.1223625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1223748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1224032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1224169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1224359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0stpc7bw
2023-01-11T22:03:10.1224558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0stpc7bw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1224835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1224945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1225224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1225359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1225545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxb615ip
2023-01-11T22:03:10.1225741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxb615ip/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1226013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1226139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1226417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1226539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1226724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2egm4k0
2023-01-11T22:03:10.1226915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2egm4k0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1227186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1227310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1227589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1227724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1227908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64ri5ddu
2023-01-11T22:03:10.1228100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64ri5ddu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1228251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1228414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1228576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1228738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1228805Z ok (2.743s)
2023-01-11T22:03:10.1228810Z 
2023-01-11T22:03:10.1229036Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1229111Z Ran 1 test in 2.744s
2023-01-11T22:03:10.1229117Z 
2023-01-11T22:03:10.1229178Z OK
2023-01-11T22:03:10.1229184Z 
2023-01-11T22:03:10.1229282Z Generating XML reports...
2023-01-11T22:03:10.1229638Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213605.xml
2023-01-11T22:03:10.1229908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1230030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1230309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1230444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1230630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxyyp8n6f
2023-01-11T22:03:10.1230826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxyyp8n6f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1231084Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1231089Z 
2023-01-11T22:03:10.1231150Z Running tests...
2023-01-11T22:03:10.1231343Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1231578Z   test_callback_with_exception (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25415
2023-01-11T22:03:10.1231737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25416
2023-01-11T22:03:10.1231890Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25417
2023-01-11T22:03:10.1232045Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25418
2023-01-11T22:03:10.1232318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1232443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1232718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1232853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1233037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cvagmns
2023-01-11T22:03:10.1233234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cvagmns/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1233506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1233629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1233909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1234049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1234233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1zy6ihw1
2023-01-11T22:03:10.1234415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1zy6ihw1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1234689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1234813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1235094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1235228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1235413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwh5kabvv
2023-01-11T22:03:10.1235715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1235837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1236130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1236266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1236455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2qg9bes
2023-01-11T22:03:10.1236653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwh5kabvv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1236842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2qg9bes/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1237008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1237175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1237340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1237502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1237557Z ok (2.744s)
2023-01-11T22:03:10.1237563Z 
2023-01-11T22:03:10.1237758Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1237835Z Ran 1 test in 2.745s
2023-01-11T22:03:10.1237841Z 
2023-01-11T22:03:10.1237901Z OK
2023-01-11T22:03:10.1237906Z 
2023-01-11T22:03:10.1237989Z Generating XML reports...
2023-01-11T22:03:10.1238335Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213610.xml
2023-01-11T22:03:10.1238610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1238734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1239089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1239228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1239414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpve22yqrg
2023-01-11T22:03:10.1239608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpve22yqrg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1239865Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1239870Z 
2023-01-11T22:03:10.1239944Z Running tests...
2023-01-11T22:03:10.1240144Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1240396Z   test_create_local_script_class_rref_in_py (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25654
2023-01-11T22:03:10.1240557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25655
2023-01-11T22:03:10.1240701Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25656
2023-01-11T22:03:10.1240858Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25657
2023-01-11T22:03:10.1241137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1241262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1241548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1241685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1241873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xd0p0ga
2023-01-11T22:03:10.1242150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1242316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1242614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1242750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1242937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8u7mmen
2023-01-11T22:03:10.1243207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1243330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1243609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1243744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1243932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4tcsp5e
2023-01-11T22:03:10.1244117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xd0p0ga/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1244308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8u7mmen/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1244496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4tcsp5e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1244768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1244893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1245172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1245307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1245491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpblf6oige
2023-01-11T22:03:10.1245686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpblf6oige/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1245840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1246002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1246165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1246331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1246400Z ok (2.643s)
2023-01-11T22:03:10.1246406Z 
2023-01-11T22:03:10.1246605Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1246679Z Ran 1 test in 2.643s
2023-01-11T22:03:10.1246684Z 
2023-01-11T22:03:10.1246744Z OK
2023-01-11T22:03:10.1246749Z 
2023-01-11T22:03:10.1246823Z Generating XML reports...
2023-01-11T22:03:10.1247170Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213615.xml
2023-01-11T22:03:10.1247447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1247571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1247850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1247985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1248169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptd39d_on
2023-01-11T22:03:10.1248363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptd39d_on/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1248615Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1248653Z 
2023-01-11T22:03:10.1248714Z Running tests...
2023-01-11T22:03:10.1248909Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1249183Z   test_create_local_script_module_rref_in_py (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25877
2023-01-11T22:03:10.1249342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25878
2023-01-11T22:03:10.1249497Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25879
2023-01-11T22:03:10.1249650Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25880
2023-01-11T22:03:10.1249926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1250049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1250331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1250456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1250643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxewdf79p
2023-01-11T22:03:10.1250839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxewdf79p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1251110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1251233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1251514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1251648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1251830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperesbiuk
2023-01-11T22:03:10.1252013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperesbiuk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1252285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1252407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1252687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1252821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1253005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppl0s7r8t
2023-01-11T22:03:10.1253197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppl0s7r8t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1253466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1253590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1253860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1253994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1254176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmz4juwtk
2023-01-11T22:03:10.1254369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmz4juwtk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1254536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1254699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1254861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1255020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1255104Z ok (2.655s)
2023-01-11T22:03:10.1255110Z 
2023-01-11T22:03:10.1255348Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1255423Z Ran 1 test in 2.655s
2023-01-11T22:03:10.1255429Z 
2023-01-11T22:03:10.1255490Z OK
2023-01-11T22:03:10.1255495Z 
2023-01-11T22:03:10.1255579Z Generating XML reports...
2023-01-11T22:03:10.1255929Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213619.xml
2023-01-11T22:03:10.1256205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1256329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1256609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1256735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1256919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperbaeavz
2023-01-11T22:03:10.1257117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperbaeavz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1257369Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1257374Z 
2023-01-11T22:03:10.1257446Z Running tests...
2023-01-11T22:03:10.1257647Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1257889Z   test_create_script_module_on_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26100
2023-01-11T22:03:10.1258048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26101
2023-01-11T22:03:10.1258190Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26102
2023-01-11T22:03:10.1258347Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26103
2023-01-11T22:03:10.1258627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1258751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1259033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1259169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1259356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6ewac1q
2023-01-11T22:03:10.1259550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6ewac1q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1259821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1259934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1260216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1260352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1260537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpaq9z6y
2023-01-11T22:03:10.1260733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpaq9z6y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1261005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1261130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1261408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1261531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1261966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90qsqqa9
2023-01-11T22:03:10.1262209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90qsqqa9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1262488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1262615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1262895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1263031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1263219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxaibcbg2
2023-01-11T22:03:10.1263415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxaibcbg2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1263568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1263736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1263901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1264064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1264133Z ok (2.752s)
2023-01-11T22:03:10.1264139Z 
2023-01-11T22:03:10.1264339Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1264416Z Ran 1 test in 2.752s
2023-01-11T22:03:10.1264421Z 
2023-01-11T22:03:10.1264483Z OK
2023-01-11T22:03:10.1264488Z 
2023-01-11T22:03:10.1264558Z Generating XML reports...
2023-01-11T22:03:10.1264910Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213624.xml
2023-01-11T22:03:10.1265190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1265318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1265603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1265742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1265927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4kqlbee
2023-01-11T22:03:10.1266122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4kqlbee/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1266376Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1266381Z 
2023-01-11T22:03:10.1266441Z Running tests...
2023-01-11T22:03:10.1266638Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1267357Z   test_future_passed_between_python_and_jit (__main__.TensorPipeJitRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84046 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.319s)
2023-01-11T22:03:10.1267364Z 
2023-01-11T22:03:10.1267561Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1267636Z Ran 1 test in 0.319s
2023-01-11T22:03:10.1267642Z 
2023-01-11T22:03:10.1267713Z OK (skipped=1)
2023-01-11T22:03:10.1267718Z 
2023-01-11T22:03:10.1267802Z Generating XML reports...
2023-01-11T22:03:10.1268149Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213629.xml
2023-01-11T22:03:10.1268424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1268593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1268892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1269034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1269220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcdd47ln9
2023-01-11T22:03:10.1269414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcdd47ln9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1269672Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1269678Z 
2023-01-11T22:03:10.1269751Z Running tests...
2023-01-11T22:03:10.1269952Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1270192Z   test_future_python_annotation (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26333
2023-01-11T22:03:10.1270355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26334
2023-01-11T22:03:10.1270500Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26335
2023-01-11T22:03:10.1270658Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26336
2023-01-11T22:03:10.1270935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1271061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1271348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1271485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1271673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpog7wn_k5
2023-01-11T22:03:10.1271871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpog7wn_k5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1272133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1272259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1272536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1272674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1272860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpseoy95um
2023-01-11T22:03:10.1273056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpseoy95um/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1273327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1273454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1273738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1273860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1274048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6du8h4q
2023-01-11T22:03:10.1274243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6du8h4q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1274515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1274640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1274921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1275056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1275271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvm_k56ma
2023-01-11T22:03:10.1275482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvm_k56ma/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1275647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1275813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1275977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1276141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1276211Z ok (2.650s)
2023-01-11T22:03:10.1276216Z 
2023-01-11T22:03:10.1276415Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1276492Z Ran 1 test in 2.650s
2023-01-11T22:03:10.1276500Z 
2023-01-11T22:03:10.1276562Z OK
2023-01-11T22:03:10.1276567Z 
2023-01-11T22:03:10.1276638Z Generating XML reports...
2023-01-11T22:03:10.1276993Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213631.xml
2023-01-11T22:03:10.1277274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1277400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1277689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1277829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1278017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqc_2a5xs
2023-01-11T22:03:10.1278215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqc_2a5xs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1278460Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1278476Z 
2023-01-11T22:03:10.1278537Z Running tests...
2023-01-11T22:03:10.1278737Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1279044Z   test_kwargs_not_passed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26560
2023-01-11T22:03:10.1279204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26561
2023-01-11T22:03:10.1279361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26562
2023-01-11T22:03:10.1279513Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26563
2023-01-11T22:03:10.1279791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1279916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1280189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1280328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1280516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmauxxoo
2023-01-11T22:03:10.1280714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmauxxoo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1280991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1281117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1281401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1281538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1281754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqn7ade_i
2023-01-11T22:03:10.1281984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqn7ade_i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1282263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1282391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1282669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1282804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1282989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzbfe09y
2023-01-11T22:03:10.1283182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzbfe09y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1283453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1283567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1283848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1283983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1284166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcc8nth65
2023-01-11T22:03:10.1284358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcc8nth65/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1284520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1284688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1284849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1285014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1285068Z ok (2.744s)
2023-01-11T22:03:10.1285073Z 
2023-01-11T22:03:10.1285274Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1285349Z Ran 1 test in 2.745s
2023-01-11T22:03:10.1285355Z 
2023-01-11T22:03:10.1285415Z OK
2023-01-11T22:03:10.1285420Z 
2023-01-11T22:03:10.1285503Z Generating XML reports...
2023-01-11T22:03:10.1285849Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213635.xml
2023-01-11T22:03:10.1286122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1286244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1286511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1286648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1286833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfobq0037
2023-01-11T22:03:10.1287025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfobq0037/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1287280Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1287286Z 
2023-01-11T22:03:10.1287356Z Running tests...
2023-01-11T22:03:10.1287551Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1287800Z   test_less_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26787
2023-01-11T22:03:10.1287958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26788
2023-01-11T22:03:10.1288101Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26789
2023-01-11T22:03:10.1288284Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26790
2023-01-11T22:03:10.1288588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1288711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1288994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1289128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1289314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrfz40yd
2023-01-11T22:03:10.1289508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrfz40yd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1289770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1289895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1290177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1290313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1290497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpov6i0829
2023-01-11T22:03:10.1290690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpov6i0829/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1290966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1291088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1291366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1291492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1291677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmoif10sh
2023-01-11T22:03:10.1291872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmoif10sh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1292143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1292267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1292546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1292680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1292865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lchic9u
2023-01-11T22:03:10.1293047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lchic9u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1293213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1293378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1293541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1293702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1293770Z ok (2.745s)
2023-01-11T22:03:10.1293776Z 
2023-01-11T22:03:10.1293974Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1294049Z Ran 1 test in 2.745s
2023-01-11T22:03:10.1294055Z 
2023-01-11T22:03:10.1294115Z OK
2023-01-11T22:03:10.1294120Z 
2023-01-11T22:03:10.1294191Z Generating XML reports...
2023-01-11T22:03:10.1294539Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213640.xml
2023-01-11T22:03:10.1294843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1294994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1295274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1295409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1295593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6atbcs_5
2023-01-11T22:03:10.1295788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6atbcs_5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1296034Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1296050Z 
2023-01-11T22:03:10.1296110Z Running tests...
2023-01-11T22:03:10.1296310Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1296556Z   test_load_script_module_with_pickled_rref (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27010
2023-01-11T22:03:10.1296712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27011
2023-01-11T22:03:10.1296865Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27012
2023-01-11T22:03:10.1297020Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27013
2023-01-11T22:03:10.1297293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1297416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1297687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1297823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1298009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrmxgyp7
2023-01-11T22:03:10.1298208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrmxgyp7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1298483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1298606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1298886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1299020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1299205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpws7ph6i9
2023-01-11T22:03:10.1299387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpws7ph6i9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1299659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1299782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1300063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1300196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1300379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ff97070
2023-01-11T22:03:10.1300567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ff97070/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1300836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1300946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1301265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1301397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1301620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1r9lfylg
2023-01-11T22:03:10.1301929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1r9lfylg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1302095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1302261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1302424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1302588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1302642Z ok (2.842s)
2023-01-11T22:03:10.1302650Z 
2023-01-11T22:03:10.1302853Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1302930Z Ran 1 test in 2.842s
2023-01-11T22:03:10.1302935Z 
2023-01-11T22:03:10.1302996Z OK
2023-01-11T22:03:10.1303002Z 
2023-01-11T22:03:10.1303088Z Generating XML reports...
2023-01-11T22:03:10.1303438Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213645.xml
2023-01-11T22:03:10.1303716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1303842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1304112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1304246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1304432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkld6rojg
2023-01-11T22:03:10.1304630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkld6rojg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1304886Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1304891Z 
2023-01-11T22:03:10.1304963Z Running tests...
2023-01-11T22:03:10.1305156Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1305388Z   test_local_rref_local_value (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27233
2023-01-11T22:03:10.1305545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27234
2023-01-11T22:03:10.1305687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27235
2023-01-11T22:03:10.1305838Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27236
2023-01-11T22:03:10.1306114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1306239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1306521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1306655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1306842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_lyxj_kq
2023-01-11T22:03:10.1307036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_lyxj_kq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1307297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1307422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1307705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1307931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1308153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6z6r513z
2023-01-11T22:03:10.1308348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6z6r513z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1308621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1308746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1309026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1309149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1309336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3efqst_
2023-01-11T22:03:10.1309532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3efqst_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1309805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1309928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1310209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1310342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1310528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7n53m_o
2023-01-11T22:03:10.1310722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7n53m_o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1310874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1311036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1311203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1311367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1311434Z ok (2.642s)
2023-01-11T22:03:10.1311439Z 
2023-01-11T22:03:10.1311636Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1311712Z Ran 1 test in 2.643s
2023-01-11T22:03:10.1311717Z 
2023-01-11T22:03:10.1311777Z OK
2023-01-11T22:03:10.1311782Z 
2023-01-11T22:03:10.1311853Z Generating XML reports...
2023-01-11T22:03:10.1312198Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213650.xml
2023-01-11T22:03:10.1312472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1312598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1312879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1313016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1313201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwx0n2_s
2023-01-11T22:03:10.1313396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwx0n2_s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1313648Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1313654Z 
2023-01-11T22:03:10.1313714Z Running tests...
2023-01-11T22:03:10.1313910Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1314156Z   test_more_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27456
2023-01-11T22:03:10.1314346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27457
2023-01-11T22:03:10.1314526Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27458
2023-01-11T22:03:10.1314682Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27459
2023-01-11T22:03:10.1314959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1315084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1315356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1315492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1315676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93d9p7mw
2023-01-11T22:03:10.1315870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93d9p7mw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1316146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1316271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1316552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1316688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1316872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5zkq_i1
2023-01-11T22:03:10.1317052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5zkq_i1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1317323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1317446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1317727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1317863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1318046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35y_jwxp
2023-01-11T22:03:10.1318238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35y_jwxp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1318510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1318621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1318979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1319118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1319305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxginvm9
2023-01-11T22:03:10.1319503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxginvm9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1319668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1319834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1319993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1320156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1320211Z ok (2.548s)
2023-01-11T22:03:10.1320216Z 
2023-01-11T22:03:10.1320415Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1320496Z Ran 1 test in 2.548s
2023-01-11T22:03:10.1320501Z 
2023-01-11T22:03:10.1320563Z OK
2023-01-11T22:03:10.1320602Z 
2023-01-11T22:03:10.1320688Z Generating XML reports...
2023-01-11T22:03:10.1321067Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213654.xml
2023-01-11T22:03:10.1321345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1321467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1321737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1321870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1322056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps691ictn
2023-01-11T22:03:10.1322249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps691ictn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1322505Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1322513Z 
2023-01-11T22:03:10.1322585Z Running tests...
2023-01-11T22:03:10.1322783Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1323018Z   test_my_script_module_with_rrefs (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27679
2023-01-11T22:03:10.1323176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27680
2023-01-11T22:03:10.1323321Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27681
2023-01-11T22:03:10.1323474Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27682
2023-01-11T22:03:10.1323750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1323875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1324160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1324295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1324484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpss8qhp2e
2023-01-11T22:03:10.1324681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpss8qhp2e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1324954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1325065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1325348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1325484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1325672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfl0beux
2023-01-11T22:03:10.1325869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfl0beux/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1326145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1326271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1326555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1326677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1326866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc30e7go6
2023-01-11T22:03:10.1327061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc30e7go6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1327336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1327492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1327806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1327942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1328128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuoq3ybop
2023-01-11T22:03:10.1328325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuoq3ybop/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1328479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1328643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1328810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1328975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1329045Z ok (2.852s)
2023-01-11T22:03:10.1329050Z 
2023-01-11T22:03:10.1329254Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1329330Z Ran 1 test in 2.852s
2023-01-11T22:03:10.1329335Z 
2023-01-11T22:03:10.1337603Z OK
2023-01-11T22:03:10.1337621Z 
2023-01-11T22:03:10.1337787Z Generating XML reports...
2023-01-11T22:03:10.1338188Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213659.xml
2023-01-11T22:03:10.1338479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1338612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1338905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1339055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1339247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbd197hz8
2023-01-11T22:03:10.1339453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbd197hz8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1339715Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1339720Z 
2023-01-11T22:03:10.1339796Z Running tests...
2023-01-11T22:03:10.1339988Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1340242Z   test_no_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27902
2023-01-11T22:03:10.1340404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27903
2023-01-11T22:03:10.1340562Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27904
2023-01-11T22:03:10.1340722Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27905
2023-01-11T22:03:10.1341004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1341133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1341422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1341561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1342017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprea_syvc
2023-01-11T22:03:10.1342245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprea_syvc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1342534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1342785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1343125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1343267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1343455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgewt6k08
2023-01-11T22:03:10.1343651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgewt6k08/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1343917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1344044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1344327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1344467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1344657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_91eeww
2023-01-11T22:03:10.1344856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_91eeww/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1345134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1345263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1345549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1345674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1345863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bmeivxn
2023-01-11T22:03:10.1346062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bmeivxn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1346232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1346401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1346566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1346733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1346805Z ok (2.639s)
2023-01-11T22:03:10.1346811Z 
2023-01-11T22:03:10.1346999Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1347078Z Ran 1 test in 2.639s
2023-01-11T22:03:10.1347083Z 
2023-01-11T22:03:10.1347146Z OK
2023-01-11T22:03:10.1347151Z 
2023-01-11T22:03:10.1347237Z Generating XML reports...
2023-01-11T22:03:10.1347591Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213703.xml
2023-01-11T22:03:10.1347877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1348005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1348295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1348433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1348610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3ukq8sf
2023-01-11T22:03:10.1348810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3ukq8sf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1349069Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1349074Z 
2023-01-11T22:03:10.1349150Z Running tests...
2023-01-11T22:03:10.1349350Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1349642Z   test_record_function_jit_end_callbacks_with_fork (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28129
2023-01-11T22:03:10.1349837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28130
2023-01-11T22:03:10.1349997Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28131
2023-01-11T22:03:10.1350141Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28132
2023-01-11T22:03:10.1350418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1350542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1350826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1350965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1351160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9xaztiu
2023-01-11T22:03:10.1351363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9xaztiu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1351637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1351765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1352035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1352175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1352359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0g3s3hyu
2023-01-11T22:03:10.1352556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0g3s3hyu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1352832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1352961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1353246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1353383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1353570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8z_j776g
2023-01-11T22:03:10.1353829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1353955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1354237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1354374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1354565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpif46zti0
2023-01-11T22:03:10.1354762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8z_j776g/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1354954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpif46zti0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1355123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1355277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1355442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1355607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1355680Z ok (3.243s)
2023-01-11T22:03:10.1355686Z 
2023-01-11T22:03:10.1355889Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1356001Z Ran 1 test in 3.243s
2023-01-11T22:03:10.1356007Z 
2023-01-11T22:03:10.1356068Z OK
2023-01-11T22:03:10.1356073Z 
2023-01-11T22:03:10.1356204Z Generating XML reports...
2023-01-11T22:03:10.1356550Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213708.xml
2023-01-11T22:03:10.1356826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1356952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1357234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1357373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1357560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnn5xv5t
2023-01-11T22:03:10.1357760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnn5xv5t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1358024Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1358029Z 
2023-01-11T22:03:10.1358105Z Running tests...
2023-01-11T22:03:10.1358294Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1358544Z   test_record_function_on_caller_rpc_async (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28184
2023-01-11T22:03:10.1358704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28185
2023-01-11T22:03:10.1358959Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28186
2023-01-11T22:03:10.1359120Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28187
2023-01-11T22:03:10.1359403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1359536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1359831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1359973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1360148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1eaz5j0l
2023-01-11T22:03:10.1360348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1eaz5j0l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1360627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1360761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1361048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1361189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1361381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmbi9na_
2023-01-11T22:03:10.1361580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmbi9na_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1361843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1361972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1362255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1362394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1362585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2epxyakk
2023-01-11T22:03:10.1362820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2epxyakk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1363125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1363254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1363533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1363657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1363844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptai98vhr
2023-01-11T22:03:10.1364042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptai98vhr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1364241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1364415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1364644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1369983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1370070Z ok (2.650s)
2023-01-11T22:03:10.1370076Z 
2023-01-11T22:03:10.1370282Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1370346Z Ran 1 test in 2.651s
2023-01-11T22:03:10.1370351Z 
2023-01-11T22:03:10.1370412Z OK
2023-01-11T22:03:10.1370417Z 
2023-01-11T22:03:10.1370502Z Generating XML reports...
2023-01-11T22:03:10.1370848Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213713.xml
2023-01-11T22:03:10.1371123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1371250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1371537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1371678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1371853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjdc4h8uv
2023-01-11T22:03:10.1372053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjdc4h8uv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1372308Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1372314Z 
2023-01-11T22:03:10.1372386Z Running tests...
2023-01-11T22:03:10.1372587Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1372820Z   test_remote_script_module (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28411
2023-01-11T22:03:10.1372982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28412
2023-01-11T22:03:10.1373138Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28413
2023-01-11T22:03:10.1393875Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28414
2023-01-11T22:03:10.1394197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1394322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1394599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1394727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1394907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pchit35
2023-01-11T22:03:10.1395095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pchit35/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1395432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1395585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1395859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1395989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1396166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9q6mg00
2023-01-11T22:03:10.1396353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9q6mg00/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1396619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1396734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1397010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1397144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1397321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jeu1lxu
2023-01-11T22:03:10.1397503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jeu1lxu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1397768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1397885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1398155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1398282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1398459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1zfypjlp
2023-01-11T22:03:10.1398649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1zfypjlp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1398810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1399063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1399214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1399370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1399431Z ok (2.742s)
2023-01-11T22:03:10.1399436Z 
2023-01-11T22:03:10.1399630Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1399698Z Ran 1 test in 2.742s
2023-01-11T22:03:10.1399704Z 
2023-01-11T22:03:10.1399759Z OK
2023-01-11T22:03:10.1399764Z 
2023-01-11T22:03:10.1399844Z Generating XML reports...
2023-01-11T22:03:10.1400190Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213718.xml
2023-01-11T22:03:10.1400455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1400574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1400848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1400975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1401154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizupp2ok
2023-01-11T22:03:10.1401346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizupp2ok/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1401595Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1401634Z 
2023-01-11T22:03:10.1401702Z Running tests...
2023-01-11T22:03:10.1401893Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1402137Z   test_remote_script_throw (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28634
2023-01-11T22:03:10.1402291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28635
2023-01-11T22:03:10.1402440Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28636
2023-01-11T22:03:10.1402589Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28637
2023-01-11T22:03:10.1402857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1402975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1403253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1403386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1403559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqjfbrl2a
2023-01-11T22:03:10.1403748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqjfbrl2a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1404015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1404136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1404409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1404538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1404715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3g6_2h4
2023-01-11T22:03:10.1404903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3g6_2h4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1405172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1405282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1405558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1405686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1405864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp319jbm8j
2023-01-11T22:03:10.1406051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp319jbm8j/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1406317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1406436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1406710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1406832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1407008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl4lojm9j
2023-01-11T22:03:10.1407196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl4lojm9j/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1407354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1407510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1407667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1407821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1407915Z ok (2.853s)
2023-01-11T22:03:10.1407921Z 
2023-01-11T22:03:10.1408114Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1408205Z Ran 1 test in 2.853s
2023-01-11T22:03:10.1408211Z 
2023-01-11T22:03:10.1408267Z OK
2023-01-11T22:03:10.1408272Z 
2023-01-11T22:03:10.1408352Z Generating XML reports...
2023-01-11T22:03:10.1408699Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213722.xml
2023-01-11T22:03:10.1408966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1409084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1409362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1409494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1409667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_dseloe
2023-01-11T22:03:10.1409859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_dseloe/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1410109Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1410115Z 
2023-01-11T22:03:10.1410182Z Running tests...
2023-01-11T22:03:10.1410371Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1410593Z   test_remote_script_udf (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28857
2023-01-11T22:03:10.1410745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28858
2023-01-11T22:03:10.1410894Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28859
2023-01-11T22:03:10.1411042Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28860
2023-01-11T22:03:10.1411309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1411429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1411705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1411835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1412014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpin1bp29w
2023-01-11T22:03:10.1412201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpin1bp29w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1412468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1412586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1412857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1412988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1413165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnb4as5q3
2023-01-11T22:03:10.1413351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnb4as5q3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1413616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1413734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1414007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1414135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1414343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwl24e88a
2023-01-11T22:03:10.1414552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwl24e88a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1414819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1414936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1415208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1415335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1415513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu6axiv_0
2023-01-11T22:03:10.1415702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu6axiv_0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1415861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1416022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1416175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1416332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1416394Z ok (2.644s)
2023-01-11T22:03:10.1416400Z 
2023-01-11T22:03:10.1416590Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1416659Z Ran 1 test in 2.644s
2023-01-11T22:03:10.1416665Z 
2023-01-11T22:03:10.1416719Z OK
2023-01-11T22:03:10.1416724Z 
2023-01-11T22:03:10.1416802Z Generating XML reports...
2023-01-11T22:03:10.1417144Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213727.xml
2023-01-11T22:03:10.1417405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1417526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1417802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1417932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1418111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwg9su8jz
2023-01-11T22:03:10.1418299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwg9su8jz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1418549Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1418555Z 
2023-01-11T22:03:10.1418618Z Running tests...
2023-01-11T22:03:10.1418812Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1419074Z   test_return_local_script_class_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29096
2023-01-11T22:03:10.1419239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29097
2023-01-11T22:03:10.1419395Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29098
2023-01-11T22:03:10.1419551Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29099
2023-01-11T22:03:10.1419825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1419937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1420219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1420355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1420540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgamikv4y
2023-01-11T22:03:10.1420768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgamikv4y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1421070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1421197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1421474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1421610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1422026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnheje3l
2023-01-11T22:03:10.1422224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnheje3l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1422504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1422630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1422911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1423048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1423233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubpyjhn1
2023-01-11T22:03:10.1423429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubpyjhn1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1423686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1423809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1424089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1424225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1424411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_5v2j952
2023-01-11T22:03:10.1424601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_5v2j952/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1424766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1424933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1425096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1425247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1425315Z ok (2.747s)
2023-01-11T22:03:10.1425321Z 
2023-01-11T22:03:10.1425522Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1425600Z Ran 1 test in 2.748s
2023-01-11T22:03:10.1425605Z 
2023-01-11T22:03:10.1425667Z OK
2023-01-11T22:03:10.1425672Z 
2023-01-11T22:03:10.1425757Z Generating XML reports...
2023-01-11T22:03:10.1426110Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213732.xml
2023-01-11T22:03:10.1426385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1426496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1426779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1426915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1427101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfi6qp1yx
2023-01-11T22:03:10.1427299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfi6qp1yx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1427619Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1427674Z 
2023-01-11T22:03:10.1427753Z Running tests...
2023-01-11T22:03:10.1427951Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1428212Z   test_return_local_script_module_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29323
2023-01-11T22:03:10.1428358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29324
2023-01-11T22:03:10.1428513Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29325
2023-01-11T22:03:10.1428667Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29326
2023-01-11T22:03:10.1428942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1429072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1429358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1429495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1429681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprksdvvnw
2023-01-11T22:03:10.1429870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprksdvvnw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1430144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1430268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1430552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1430691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1430877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37tm9vx9
2023-01-11T22:03:10.1431075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37tm9vx9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1431347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1431471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1431738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1431874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1432059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdldqir5
2023-01-11T22:03:10.1432255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdldqir5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1432532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1432660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1432940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1433076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1433261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4ckdy_d
2023-01-11T22:03:10.1433448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4ckdy_d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1433614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1433780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1433976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1434165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1434235Z ok (2.952s)
2023-01-11T22:03:10.1434241Z 
2023-01-11T22:03:10.1434441Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1434520Z Ran 1 test in 2.952s
2023-01-11T22:03:10.1434525Z 
2023-01-11T22:03:10.1434573Z OK
2023-01-11T22:03:10.1434579Z 
2023-01-11T22:03:10.1434664Z Generating XML reports...
2023-01-11T22:03:10.1435015Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213736.xml
2023-01-11T22:03:10.1435288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1435414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1435699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1435838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1436024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqok5gquq
2023-01-11T22:03:10.1436221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqok5gquq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1436462Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1436467Z 
2023-01-11T22:03:10.1436541Z Running tests...
2023-01-11T22:03:10.1436740Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1436976Z   test_rpc_async_jit_profiled (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29550
2023-01-11T22:03:10.1437133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29551
2023-01-11T22:03:10.1437291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29552
2023-01-11T22:03:10.1437447Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29553
2023-01-11T22:03:10.1437722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1437834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1438119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1438256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1438446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyf4jrgdp
2023-01-11T22:03:10.1438642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyf4jrgdp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1438994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1439124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1439406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1439542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1439715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpycnnrx8x
2023-01-11T22:03:10.1439911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpycnnrx8x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1440186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1440310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1440587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1440775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1441022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpae5fj45x
2023-01-11T22:03:10.1441218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpae5fj45x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1441477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1441600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1441880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1442014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1442199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx6rh8t99
2023-01-11T22:03:10.1442397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx6rh8t99/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1442564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1442729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1442892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1443043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1443110Z ok (2.642s)
2023-01-11T22:03:10.1443116Z 
2023-01-11T22:03:10.1443315Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1443390Z Ran 1 test in 2.643s
2023-01-11T22:03:10.1443395Z 
2023-01-11T22:03:10.1443455Z OK
2023-01-11T22:03:10.1443460Z 
2023-01-11T22:03:10.1443543Z Generating XML reports...
2023-01-11T22:03:10.1443898Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213741.xml
2023-01-11T22:03:10.1444175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1444287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1444568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1444704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1444891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpji4if640
2023-01-11T22:03:10.1445086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpji4if640/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1445339Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1445347Z 
2023-01-11T22:03:10.1445419Z Running tests...
2023-01-11T22:03:10.1445615Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1445861Z   test_rpc_torchscript_record_function (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29777
2023-01-11T22:03:10.1446006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29778
2023-01-11T22:03:10.1446161Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29779
2023-01-11T22:03:10.1446316Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29780
2023-01-11T22:03:10.1446594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1446719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1447000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1447169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1447382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrecth4b
2023-01-11T22:03:10.1447568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrecth4b/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1447842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1447966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1448247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1448383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1448568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyel4ujy
2023-01-11T22:03:10.1448769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyel4ujy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1449046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1449170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1449440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1449575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1449759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnocc7eiy
2023-01-11T22:03:10.1449956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnocc7eiy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1450228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1450355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1450637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1450776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1450960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnma0loam
2023-01-11T22:03:10.1451142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnma0loam/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1451306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1451469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1451632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1451793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1451864Z ok (2.750s)
2023-01-11T22:03:10.1451870Z 
2023-01-11T22:03:10.1452067Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1452145Z Ran 1 test in 2.750s
2023-01-11T22:03:10.1452150Z 
2023-01-11T22:03:10.1452197Z OK
2023-01-11T22:03:10.1452202Z 
2023-01-11T22:03:10.1452287Z Generating XML reports...
2023-01-11T22:03:10.1452638Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213746.xml
2023-01-11T22:03:10.1452916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1453042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1453326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1453462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1453680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7vtk9ek
2023-01-11T22:03:10.1453905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7vtk9ek/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1454148Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1454154Z 
2023-01-11T22:03:10.1454228Z Running tests...
2023-01-11T22:03:10.1454427Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1454657Z   test_rref_as_arg_and_return (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30004
2023-01-11T22:03:10.1454817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30005
2023-01-11T22:03:10.1454971Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30006
2023-01-11T22:03:10.1455125Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30007
2023-01-11T22:03:10.1455407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1455520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1455801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1455935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1456122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjw1fauyi
2023-01-11T22:03:10.1456319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjw1fauyi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1456589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1456714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1456998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1457137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1457307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bsf55s6
2023-01-11T22:03:10.1457502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bsf55s6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1457776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1457900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1458181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1458317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1458504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6i866ydp
2023-01-11T22:03:10.1458698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6i866ydp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1458961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1459085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1459365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1459499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1459684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42alfhg2
2023-01-11T22:03:10.1459879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42alfhg2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1460045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1460243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1460435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1460586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1460655Z ok (2.739s)
2023-01-11T22:03:10.1460661Z 
2023-01-11T22:03:10.1460860Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1460937Z Ran 1 test in 2.739s
2023-01-11T22:03:10.1460942Z 
2023-01-11T22:03:10.1461002Z OK
2023-01-11T22:03:10.1461008Z 
2023-01-11T22:03:10.1461091Z Generating XML reports...
2023-01-11T22:03:10.1461446Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213751.xml
2023-01-11T22:03:10.1461867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1461983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1462268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1462404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1462590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2up4x3o
2023-01-11T22:03:10.1462787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2up4x3o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1463042Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1463048Z 
2023-01-11T22:03:10.1463124Z Running tests...
2023-01-11T22:03:10.1463324Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1463546Z   test_rref_is_owner (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30227
2023-01-11T22:03:10.1463692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30228
2023-01-11T22:03:10.1463850Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30229
2023-01-11T22:03:10.1464004Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30230
2023-01-11T22:03:10.1464279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1464403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1464685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1464820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1465007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdh6vxxd
2023-01-11T22:03:10.1465282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1465396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1465674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1465812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1466002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn6uyt4v6
2023-01-11T22:03:10.1466201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdh6vxxd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1466394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn6uyt4v6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1466665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1466849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1467151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1467289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1467473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2649brnn
2023-01-11T22:03:10.1467665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2649brnn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1467936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1468060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1468339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1468474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1468661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphd8bl64j
2023-01-11T22:03:10.1468842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphd8bl64j/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1469008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1469173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1469337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1469499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1469568Z ok (2.845s)
2023-01-11T22:03:10.1469574Z 
2023-01-11T22:03:10.1469772Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1469848Z Ran 1 test in 2.845s
2023-01-11T22:03:10.1469857Z 
2023-01-11T22:03:10.1469905Z OK
2023-01-11T22:03:10.1469910Z 
2023-01-11T22:03:10.1469996Z Generating XML reports...
2023-01-11T22:03:10.1470352Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213755.xml
2023-01-11T22:03:10.1470627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1470753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1471035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1471170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1471372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphf_i6zj9
2023-01-11T22:03:10.1471622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphf_i6zj9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1471872Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1471878Z 
2023-01-11T22:03:10.1471951Z Running tests...
2023-01-11T22:03:10.1472151Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1472391Z   test_rref_jit_pickle_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30450
2023-01-11T22:03:10.1472551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30451
2023-01-11T22:03:10.1472707Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30452
2023-01-11T22:03:10.1472861Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30453
2023-01-11T22:03:10.1473138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1473250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1473577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1473756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1473943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplwk1j6jv
2023-01-11T22:03:10.1474139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplwk1j6jv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1474413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1474539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1474821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1474956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1475129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0lq3i_ar
2023-01-11T22:03:10.1475325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0lq3i_ar/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1475596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1475720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1476003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1476138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1476323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_nprakxn
2023-01-11T22:03:10.1476519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_nprakxn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1476782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1476906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1477187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1477323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1477507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0z4_aj40
2023-01-11T22:03:10.1477702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0z4_aj40/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1477867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1478031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1478194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1478345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1478412Z ok (2.746s)
2023-01-11T22:03:10.1478420Z 
2023-01-11T22:03:10.1478620Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1478696Z Ran 1 test in 2.746s
2023-01-11T22:03:10.1478702Z 
2023-01-11T22:03:10.1478763Z OK
2023-01-11T22:03:10.1478768Z 
2023-01-11T22:03:10.1478918Z Generating XML reports...
2023-01-11T22:03:10.1479273Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213800.xml
2023-01-11T22:03:10.1479550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1479661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1479942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1480115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1480330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28e7ikik
2023-01-11T22:03:10.1480530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28e7ikik/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1480786Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1480791Z 
2023-01-11T22:03:10.1480865Z Running tests...
2023-01-11T22:03:10.1481063Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1481289Z   test_rref_list_mutate (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30673
2023-01-11T22:03:10.1481432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30674
2023-01-11T22:03:10.1481590Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30675
2023-01-11T22:03:10.1481744Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30676
2023-01-11T22:03:10.1482024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1482150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1482435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1482571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1482758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytje4r9c
2023-01-11T22:03:10.1482955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytje4r9c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1483214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1483342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1483625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1483763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1483949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmc9rusa
2023-01-11T22:03:10.1484145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmc9rusa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1484419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1484543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1484810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1484947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1485137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcllq6bl
2023-01-11T22:03:10.1485332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcllq6bl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1485602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1485728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1486007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1486143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1486327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2iiqrzq9
2023-01-11T22:03:10.1486508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2iiqrzq9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1486712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1486906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1487071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1487233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1487301Z ok (2.850s)
2023-01-11T22:03:10.1487307Z 
2023-01-11T22:03:10.1487506Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1487581Z Ran 1 test in 2.850s
2023-01-11T22:03:10.1487586Z 
2023-01-11T22:03:10.1487634Z OK
2023-01-11T22:03:10.1487639Z 
2023-01-11T22:03:10.1487725Z Generating XML reports...
2023-01-11T22:03:10.1488075Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213805.xml
2023-01-11T22:03:10.1488351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1488479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1488757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1488892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1489079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25evy7fq
2023-01-11T22:03:10.1489277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25evy7fq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1489521Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1489526Z 
2023-01-11T22:03:10.1489600Z Running tests...
2023-01-11T22:03:10.1489801Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1490028Z   test_rref_local_value (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30896
2023-01-11T22:03:10.1490185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30897
2023-01-11T22:03:10.1490340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30898
2023-01-11T22:03:10.1490494Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30899
2023-01-11T22:03:10.1490769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1490882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1491163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1491298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1491487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa3go9v3o
2023-01-11T22:03:10.1491683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa3go9v3o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1491957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1492082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1492364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1492499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1492669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwm_c3m3
2023-01-11T22:03:10.1492862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwm_c3m3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1493172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1493326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1493613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1493750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1493935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplyo0baxg
2023-01-11T22:03:10.1494131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplyo0baxg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1494392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1494518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1494796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1494934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1495121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vhruhs2
2023-01-11T22:03:10.1495314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vhruhs2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1495482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1495648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1495811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1495960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1496029Z ok (2.757s)
2023-01-11T22:03:10.1496034Z 
2023-01-11T22:03:10.1496238Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1496314Z Ran 1 test in 2.758s
2023-01-11T22:03:10.1496320Z 
2023-01-11T22:03:10.1496380Z OK
2023-01-11T22:03:10.1496387Z 
2023-01-11T22:03:10.1496472Z Generating XML reports...
2023-01-11T22:03:10.1496823Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213809.xml
2023-01-11T22:03:10.1497100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1497212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1497494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1497631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1497817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1u1fwl62
2023-01-11T22:03:10.1498013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1u1fwl62/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1498268Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1498274Z 
2023-01-11T22:03:10.1498347Z Running tests...
2023-01-11T22:03:10.1498545Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1498778Z   test_rref_python_annotation (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31119
2023-01-11T22:03:10.1498923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31120
2023-01-11T22:03:10.1499077Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31121
2023-01-11T22:03:10.1499229Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31122
2023-01-11T22:03:10.1499506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1499666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1499978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1500116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1500303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwk6ptw7
2023-01-11T22:03:10.1500501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwk6ptw7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1500762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1500886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1501167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1501305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1501491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_17wk9ax
2023-01-11T22:03:10.1501807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_17wk9ax/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1502147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1502272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1502544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1502679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1502863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_l911vve
2023-01-11T22:03:10.1503062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_l911vve/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1503338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1503464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1503743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1503878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1504064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6i6vp7yo
2023-01-11T22:03:10.1504246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6i6vp7yo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1504413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1504577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1504743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1504908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1504980Z ok (2.746s)
2023-01-11T22:03:10.1504988Z 
2023-01-11T22:03:10.1505187Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1505263Z Ran 1 test in 2.746s
2023-01-11T22:03:10.1505268Z 
2023-01-11T22:03:10.1505316Z OK
2023-01-11T22:03:10.1505321Z 
2023-01-11T22:03:10.1505405Z Generating XML reports...
2023-01-11T22:03:10.1505756Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213814.xml
2023-01-11T22:03:10.1506032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1506158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1506508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1506682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1506870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4vhx6fmv
2023-01-11T22:03:10.1507065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4vhx6fmv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1507309Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1507315Z 
2023-01-11T22:03:10.1507389Z Running tests...
2023-01-11T22:03:10.1507588Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1507834Z   test_some_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31342
2023-01-11T22:03:10.1507996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31343
2023-01-11T22:03:10.1508153Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31344
2023-01-11T22:03:10.1508307Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31345
2023-01-11T22:03:10.1508583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1508694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1508978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1509117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1509306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyfxlj9o5
2023-01-11T22:03:10.1509502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyfxlj9o5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1509778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1509903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1510184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1510319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1510489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kgnygsd
2023-01-11T22:03:10.1510684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kgnygsd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1510953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1511076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1511360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1511497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1511685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpltx3surg
2023-01-11T22:03:10.1511879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpltx3surg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1512147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1512257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1512539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1512675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1512898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpktwfa_mk
2023-01-11T22:03:10.1513124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpktwfa_mk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1513292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1513456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1513619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1513766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1513834Z ok (2.740s)
2023-01-11T22:03:10.1513839Z 
2023-01-11T22:03:10.1514040Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1514116Z Ran 1 test in 2.740s
2023-01-11T22:03:10.1514121Z 
2023-01-11T22:03:10.1514183Z OK
2023-01-11T22:03:10.1514190Z 
2023-01-11T22:03:10.1514275Z Generating XML reports...
2023-01-11T22:03:10.1514623Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213819.xml
2023-01-11T22:03:10.1514901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1515026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1515297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1515433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1515617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qngzf8d
2023-01-11T22:03:10.1515816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qngzf8d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1516073Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1516080Z 
2023-01-11T22:03:10.1516154Z Running tests...
2023-01-11T22:03:10.1516354Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1516587Z   test_torchscript_function (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31569
2023-01-11T22:03:10.1516733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31570
2023-01-11T22:03:10.1516888Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31571
2023-01-11T22:03:10.1517041Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31572
2023-01-11T22:03:10.1517314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1517438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1517722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1517860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1518048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_2stewv
2023-01-11T22:03:10.1518243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_2stewv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1518502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1518625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1518988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1519125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1519310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfaki6lyy
2023-01-11T22:03:10.1519543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfaki6lyy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1519862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1519987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1520254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1520390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1520572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tw804i4
2023-01-11T22:03:10.1520763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tw804i4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1521033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1521160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1521443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1521578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1521762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbd01c3f
2023-01-11T22:03:10.1521943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbd01c3f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1522108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1522271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1522434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1522597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1522668Z ok (2.748s)
2023-01-11T22:03:10.1522674Z 
2023-01-11T22:03:10.1522876Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1522952Z Ran 1 test in 2.748s
2023-01-11T22:03:10.1522957Z 
2023-01-11T22:03:10.1523004Z OK
2023-01-11T22:03:10.1523022Z 
2023-01-11T22:03:10.1523093Z Generating XML reports...
2023-01-11T22:03:10.1523442Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213824.xml
2023-01-11T22:03:10.1523714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1523839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1524118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1524256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1524441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9z3fl3k0
2023-01-11T22:03:10.1524635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9z3fl3k0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1524878Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1524883Z 
2023-01-11T22:03:10.1524957Z Running tests...
2023-01-11T22:03:10.1525155Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1525399Z   test_torchscript_function_exception (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31792
2023-01-11T22:03:10.1525556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31793
2023-01-11T22:03:10.1525712Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31794
2023-01-11T22:03:10.1525897Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31795
2023-01-11T22:03:10.1526202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1526328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1526596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1526732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1526918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0g4a9fn2
2023-01-11T22:03:10.1527113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0g4a9fn2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1527385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1527511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1527791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1527929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1528098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvcmyz5z
2023-01-11T22:03:10.1528294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvcmyz5z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1528564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1528688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1528971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1529105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1529293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg59rw6bh
2023-01-11T22:03:10.1529490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg59rw6bh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1529767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1529878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1530159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1530296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1530481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpklbpie6z
2023-01-11T22:03:10.1530677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpklbpie6z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1530845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1531011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1531175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1531325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1531393Z ok (2.743s)
2023-01-11T22:03:10.1531398Z 
2023-01-11T22:03:10.1531599Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1531675Z Ran 1 test in 2.743s
2023-01-11T22:03:10.1531680Z 
2023-01-11T22:03:10.1531741Z OK
2023-01-11T22:03:10.1531746Z 
2023-01-11T22:03:10.1531829Z Generating XML reports...
2023-01-11T22:03:10.1532174Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213828.xml
2023-01-11T22:03:10.1532488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1532613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1532910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1533048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1533235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vbgzyk5
2023-01-11T22:03:10.1533431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vbgzyk5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1533686Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1533692Z 
2023-01-11T22:03:10.1533767Z Running tests...
2023-01-11T22:03:10.1533964Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1534214Z   test_torchscript_functions_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32015
2023-01-11T22:03:10.1534361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32016
2023-01-11T22:03:10.1534519Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32017
2023-01-11T22:03:10.1534674Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32018
2023-01-11T22:03:10.1534950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1535076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1535358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1535494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1535683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnh7wk56w
2023-01-11T22:03:10.1535881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnh7wk56w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1536142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1536267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1536549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1536685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1536870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprort6yqn
2023-01-11T22:03:10.1537069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprort6yqn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1537343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1537470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1537739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1537875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1538059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdn8k679q
2023-01-11T22:03:10.1538253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdn8k679q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1538524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1538649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1538931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1539098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1539310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps50ovozw
2023-01-11T22:03:10.1539492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps50ovozw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1539660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1539823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1539986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1540148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1540328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.1540510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.1540692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.1540995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.1541154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.1541457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.1541889Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.1542188Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.1542258Z ok (2.745s)
2023-01-11T22:03:10.1542266Z 
2023-01-11T22:03:10.1542465Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1542540Z Ran 1 test in 2.746s
2023-01-11T22:03:10.1542546Z 
2023-01-11T22:03:10.1542609Z OK
2023-01-11T22:03:10.1542613Z 
2023-01-11T22:03:10.1542697Z Generating XML reports...
2023-01-11T22:03:10.1543037Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213833.xml
2023-01-11T22:03:10.1543312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1543438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1543722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1543861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1544054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5asfi9gx
2023-01-11T22:03:10.1544256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5asfi9gx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1544514Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1544519Z 
2023-01-11T22:03:10.1544581Z Running tests...
2023-01-11T22:03:10.1544780Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1545024Z   test_unexepected_kwarg_is_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32250
2023-01-11T22:03:10.1545180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32251
2023-01-11T22:03:10.1545336Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32252
2023-01-11T22:03:10.1545491Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32253
2023-01-11T22:03:10.1545876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1546037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1546325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1546449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1546637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgkx4l7ly
2023-01-11T22:03:10.1546831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgkx4l7ly/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1547105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1547231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1547515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1547655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1547841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpru5nau2d
2023-01-11T22:03:10.1548025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpru5nau2d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1548298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1548420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1548702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1548838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1549022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpofko57mr
2023-01-11T22:03:10.1549219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpofko57mr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1549494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1549619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1549884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1550018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1550201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02r1cqgv
2023-01-11T22:03:10.1550396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02r1cqgv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1550561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1550725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1550890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1551052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1551107Z ok (2.755s)
2023-01-11T22:03:10.1551125Z 
2023-01-11T22:03:10.1551311Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1551386Z Ran 1 test in 2.755s
2023-01-11T22:03:10.1551391Z 
2023-01-11T22:03:10.1551450Z OK
2023-01-11T22:03:10.1551456Z 
2023-01-11T22:03:10.1551539Z Generating XML reports...
2023-01-11T22:03:10.1551886Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213838.xml
2023-01-11T22:03:10.1552161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1552324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1552636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1552760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1552945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lubscpo
2023-01-11T22:03:10.1553142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lubscpo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1553399Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1553405Z 
2023-01-11T22:03:10.1553478Z Running tests...
2023-01-11T22:03:10.1553675Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1553909Z   test_user_rrefs_confirmed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32473
2023-01-11T22:03:10.1554070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32474
2023-01-11T22:03:10.1554227Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32475
2023-01-11T22:03:10.1554372Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32476
2023-01-11T22:03:10.1554646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1554771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1555052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1555188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1555374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1pz6wsj
2023-01-11T22:03:10.1555647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1555772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1556045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1556181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1556364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_5gm10la
2023-01-11T22:03:10.1556561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1pz6wsj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1556752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_5gm10la/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1557029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1557156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1557437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1557574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1557745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprv1si83w
2023-01-11T22:03:10.1557942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprv1si83w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1558214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1558339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1558622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1558756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1559050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpie8_m19h
2023-01-11T22:03:10.1559276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpie8_m19h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1559430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1559596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1559758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1559921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1559990Z ok (2.848s)
2023-01-11T22:03:10.1559996Z 
2023-01-11T22:03:10.1560197Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1560272Z Ran 1 test in 2.848s
2023-01-11T22:03:10.1560278Z 
2023-01-11T22:03:10.1560342Z OK
2023-01-11T22:03:10.1560347Z 
2023-01-11T22:03:10.1560435Z Generating XML reports...
2023-01-11T22:03:10.1560773Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213842.xml
2023-01-11T22:03:10.1561046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1561170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1561453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1561590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1561775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47br_xv_
2023-01-11T22:03:10.1561967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47br_xv_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1562224Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1562229Z 
2023-01-11T22:03:10.1562301Z Running tests...
2023-01-11T22:03:10.1562487Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1562726Z   test_user_rrefs_confirmed_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32696
2023-01-11T22:03:10.1562884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32697
2023-01-11T22:03:10.1563040Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32698
2023-01-11T22:03:10.1563193Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32699
2023-01-11T22:03:10.1563468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1563592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1563877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1564003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1564190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrq3uhvk
2023-01-11T22:03:10.1564462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1564585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1564868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1565003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1565190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwxyjryc
2023-01-11T22:03:10.1565386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrq3uhvk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1565604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwxyjryc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1565923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1566048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1566328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1566463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1566647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5epetfky
2023-01-11T22:03:10.1566920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1567043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1567325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1567449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1567632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4eb1ryht
2023-01-11T22:03:10.1567826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5epetfky/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1568015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4eb1ryht/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1568181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1568346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1568510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1568675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1568730Z ok (2.744s)
2023-01-11T22:03:10.1568748Z 
2023-01-11T22:03:10.1568937Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1569012Z Ran 1 test in 2.744s
2023-01-11T22:03:10.1569017Z 
2023-01-11T22:03:10.1569077Z OK
2023-01-11T22:03:10.1569083Z 
2023-01-11T22:03:10.1569166Z Generating XML reports...
2023-01-11T22:03:10.1569515Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213847.xml
2023-01-11T22:03:10.1569791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1569914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1570197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1570322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1570509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptalhi306
2023-01-11T22:03:10.1570704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptalhi306/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1570961Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1570966Z 
2023-01-11T22:03:10.1571040Z Running tests...
2023-01-11T22:03:10.1571235Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1571499Z   test_batch_updating_parameter_server (__main__.TensorPipeParameterServerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 451
2023-01-11T22:03:10.1571656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 452
2023-01-11T22:03:10.1571808Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 453
2023-01-11T22:03:10.1571984Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 454
2023-01-11T22:03:10.1572290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1572416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1572699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1572836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1573021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprftr0adf
2023-01-11T22:03:10.1573216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprftr0adf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1573489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1573602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1573883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1574017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1574201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeaanlw6f
2023-01-11T22:03:10.1574398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeaanlw6f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1574671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1574796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1575077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1575215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1575387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8o64fwe4
2023-01-11T22:03:10.1575582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8o64fwe4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1575853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1575976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1576257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1576391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1576575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gm55r2e
2023-01-11T22:03:10.1576770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gm55r2e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1576926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1577093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1577255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1577412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1577488Z 21:38:54 Start training
2023-01-11T22:03:10.1577567Z 21:38:54 PS got 0/3 updates
2023-01-11T22:03:10.1577644Z 21:38:54 PS got 1/3 updates
2023-01-11T22:03:10.1577717Z 21:38:54 PS got 2/3 updates
2023-01-11T22:03:10.1577781Z 21:38:54 PS updated model
2023-01-11T22:03:10.1577857Z 21:38:54 PS got 0/3 updates
2023-01-11T22:03:10.1577930Z 21:38:54 PS got 1/3 updates
2023-01-11T22:03:10.1578003Z 21:38:54 PS got 2/3 updates
2023-01-11T22:03:10.1578080Z 21:38:54 PS updated model
2023-01-11T22:03:10.1578195Z 21:38:54 PS got 0/3 updates
2023-01-11T22:03:10.1578270Z 21:38:54 PS got 1/3 updates
2023-01-11T22:03:10.1578330Z 21:38:54 PS got 2/3 updates
2023-01-11T22:03:10.1578433Z 21:38:54 PS updated model
2023-01-11T22:03:10.1578510Z 21:38:54 PS got 0/3 updates
2023-01-11T22:03:10.1578582Z 21:38:54 PS got 1/3 updates
2023-01-11T22:03:10.1578655Z 21:38:54 PS got 2/3 updates
2023-01-11T22:03:10.1578730Z 21:38:54 PS updated model
2023-01-11T22:03:10.1578793Z 21:38:54 Finish training
2023-01-11T22:03:10.1578897Z 21:38:54 Time spent training: 0.01646021999476943s
2023-01-11T22:03:10.1578989Z 21:38:54 worker2 processing one batch
2023-01-11T22:03:10.1579075Z 21:38:54 worker2 reporting grads
2023-01-11T22:03:10.1579161Z 21:38:54 worker2 got updated model
2023-01-11T22:03:10.1579253Z 21:38:54 worker2 processing one batch
2023-01-11T22:03:10.1579337Z 21:38:54 worker2 reporting grads
2023-01-11T22:03:10.1579411Z 21:38:54 worker2 got updated model
2023-01-11T22:03:10.1579501Z 21:38:54 worker2 processing one batch
2023-01-11T22:03:10.1579588Z 21:38:54 worker2 reporting grads
2023-01-11T22:03:10.1579674Z 21:38:54 worker2 got updated model
2023-01-11T22:03:10.1579764Z 21:38:54 worker2 processing one batch
2023-01-11T22:03:10.1579851Z 21:38:54 worker2 reporting grads
2023-01-11T22:03:10.1579936Z 21:38:54 worker2 got updated model
2023-01-11T22:03:10.1580012Z 21:38:54 worker1 processing one batch
2023-01-11T22:03:10.1580096Z 21:38:54 worker1 reporting grads
2023-01-11T22:03:10.1580180Z 21:38:54 worker1 got updated model
2023-01-11T22:03:10.1580269Z 21:38:54 worker1 processing one batch
2023-01-11T22:03:10.1580353Z 21:38:54 worker1 reporting grads
2023-01-11T22:03:10.1580437Z 21:38:54 worker1 got updated model
2023-01-11T22:03:10.1580524Z 21:38:54 worker1 processing one batch
2023-01-11T22:03:10.1580594Z 21:38:54 worker1 reporting grads
2023-01-11T22:03:10.1580680Z 21:38:54 worker1 got updated model
2023-01-11T22:03:10.1580768Z 21:38:54 worker1 processing one batch
2023-01-11T22:03:10.1580851Z 21:38:54 worker1 reporting grads
2023-01-11T22:03:10.1580939Z 21:38:54 worker1 got updated model
2023-01-11T22:03:10.1581027Z 21:38:54 worker3 processing one batch
2023-01-11T22:03:10.1581110Z 21:38:54 worker3 reporting grads
2023-01-11T22:03:10.1581183Z 21:38:54 worker3 got updated model
2023-01-11T22:03:10.1581271Z 21:38:54 worker3 processing one batch
2023-01-11T22:03:10.1581354Z 21:38:54 worker3 reporting grads
2023-01-11T22:03:10.1581439Z 21:38:54 worker3 got updated model
2023-01-11T22:03:10.1581526Z 21:38:54 worker3 processing one batch
2023-01-11T22:03:10.1581608Z 21:38:54 worker3 reporting grads
2023-01-11T22:03:10.1581826Z 21:38:54 worker3 got updated model
2023-01-11T22:03:10.1581947Z 21:38:54 worker3 processing one batch
2023-01-11T22:03:10.1582045Z 21:38:54 worker3 reporting grads
2023-01-11T22:03:10.1582130Z 21:38:54 worker3 got updated model
2023-01-11T22:03:10.1582197Z ok (2.648s)
2023-01-11T22:03:10.1582204Z 
2023-01-11T22:03:10.1582410Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1582490Z Ran 1 test in 2.648s
2023-01-11T22:03:10.1582496Z 
2023-01-11T22:03:10.1582556Z OK
2023-01-11T22:03:10.1582561Z 
2023-01-11T22:03:10.1582632Z Generating XML reports...
2023-01-11T22:03:10.1583021Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeParameterServerTest-20230111213852.xml
2023-01-11T22:03:10.1583301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1583426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1583709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1583846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1584031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmna38dw
2023-01-11T22:03:10.1584229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmna38dw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1584559Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1584600Z 
2023-01-11T22:03:10.1584662Z Running tests...
2023-01-11T22:03:10.1584861Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1585117Z   test_rl_rpc (__main__.TensorPipeReinforcementLearningRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 676
2023-01-11T22:03:10.1585275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 677
2023-01-11T22:03:10.1585428Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 678
2023-01-11T22:03:10.1585580Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 679
2023-01-11T22:03:10.1585855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1585983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1586268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1586390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1586576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8oro1t9w
2023-01-11T22:03:10.1586770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8oro1t9w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1587043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1587167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1587447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1587585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1587771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi81qr03r
2023-01-11T22:03:10.1587953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi81qr03r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1588226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1588351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1588631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1588766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1588949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4sui02gf
2023-01-11T22:03:10.1589144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4sui02gf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1589418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1589544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1589810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1589943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1590128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafajdka4
2023-01-11T22:03:10.1590323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafajdka4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1590489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1590653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1590857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1591021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1591140Z Solved! Running reward is now 489.8051609821906!
2023-01-11T22:03:10.1591210Z ok (2.949s)
2023-01-11T22:03:10.1591216Z 
2023-01-11T22:03:10.1591415Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1591491Z Ran 1 test in 2.949s
2023-01-11T22:03:10.1591497Z 
2023-01-11T22:03:10.1591559Z OK
2023-01-11T22:03:10.1591564Z 
2023-01-11T22:03:10.1591648Z Generating XML reports...
2023-01-11T22:03:10.1592065Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeReinforcementLearningRpcTest-20230111213856.xml
2023-01-11T22:03:10.1592341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1592469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1592737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1592873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1593059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwip2h5yg
2023-01-11T22:03:10.1593254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwip2h5yg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1593512Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1593517Z 
2023-01-11T22:03:10.1593591Z Running tests...
2023-01-11T22:03:10.1593789Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1594021Z   test_bad_module (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 902
2023-01-11T22:03:10.1594165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 903
2023-01-11T22:03:10.1594446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1594571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1594853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1594989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1595175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmpd69kv
2023-01-11T22:03:10.1595371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmpd69kv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1595642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1595772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1596039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1596175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1596359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgaa114l
2023-01-11T22:03:10.1596553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgaa114l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1596716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1596880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1596973Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.1597406Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.")
2023-01-11T22:03:10.1597532Z Traceback (most recent call last):
2023-01-11T22:03:10.1597810Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1597947Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1598223Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 93, in _create_module
2023-01-11T22:03:10.1598369Z     "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "
2023-01-11T22:03:10.1598797Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.
2023-01-11T22:03:10.1598803Z 
2023-01-11T22:03:10.1598953Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.1600452Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class \'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule\'>.")\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 93, in _create_module\n    "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "\nValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class \'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule\'>.\n')
2023-01-11T22:03:10.1600552Z Traceback (most recent call last):
2023-01-11T22:03:10.1600815Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1600953Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1601109Z   File "/tmp/tmpwip2h5yg/_remote_module_non_scriptable.py", line 47, in _remote_forward
2023-01-11T22:03:10.1601204Z     module = module_rref.local_value()
2023-01-11T22:03:10.1601460Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 234, in _handle_exception
2023-01-11T22:03:10.1601529Z     raise exc
2023-01-11T22:03:10.1601640Z ValueError: On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.1602073Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.")
2023-01-11T22:03:10.1602166Z Traceback (most recent call last):
2023-01-11T22:03:10.1602428Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1602564Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1602841Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 93, in _create_module
2023-01-11T22:03:10.1602974Z     "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "
2023-01-11T22:03:10.1603395Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.
2023-01-11T22:03:10.1603400Z 
2023-01-11T22:03:10.1603405Z 
2023-01-11T22:03:10.1603498Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.1603922Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.")
2023-01-11T22:03:10.1604015Z Traceback (most recent call last):
2023-01-11T22:03:10.1604312Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1604479Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1604757Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 93, in _create_module
2023-01-11T22:03:10.1604904Z     "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "
2023-01-11T22:03:10.1605322Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.
2023-01-11T22:03:10.1605329Z 
2023-01-11T22:03:10.1605421Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.1606906Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class \'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule\'>.")\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 93, in _create_module\n    "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "\nValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class \'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule\'>.\n')
2023-01-11T22:03:10.1607004Z Traceback (most recent call last):
2023-01-11T22:03:10.1607267Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1607389Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1607545Z   File "/tmp/tmpwip2h5yg/_remote_module_non_scriptable.py", line 47, in _remote_forward
2023-01-11T22:03:10.1607640Z     module = module_rref.local_value()
2023-01-11T22:03:10.1607912Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 234, in _handle_exception
2023-01-11T22:03:10.1607981Z     raise exc
2023-01-11T22:03:10.1608092Z ValueError: On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.1608517Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.")
2023-01-11T22:03:10.1608609Z Traceback (most recent call last):
2023-01-11T22:03:10.1608855Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1608989Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1609261Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 93, in _create_module
2023-01-11T22:03:10.1609408Z     "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "
2023-01-11T22:03:10.1609827Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.
2023-01-11T22:03:10.1609834Z 
2023-01-11T22:03:10.1609839Z 
2023-01-11T22:03:10.1609907Z ok (2.438s)
2023-01-11T22:03:10.1609913Z 
2023-01-11T22:03:10.1610114Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1610191Z Ran 1 test in 2.438s
2023-01-11T22:03:10.1610197Z 
2023-01-11T22:03:10.1610257Z OK
2023-01-11T22:03:10.1610263Z 
2023-01-11T22:03:10.1610335Z Generating XML reports...
2023-01-11T22:03:10.1610705Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213901.xml
2023-01-11T22:03:10.1611014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1611183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1611469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1611608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1611795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdt1vmdao
2023-01-11T22:03:10.1611994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdt1vmdao/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1612250Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1612255Z 
2023-01-11T22:03:10.1612315Z Running tests...
2023-01-11T22:03:10.1612511Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1612747Z   test_forward_async (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1019
2023-01-11T22:03:10.1612906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1020
2023-01-11T22:03:10.1613182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1613308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1613592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1613729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1613917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpff9p4o_f
2023-01-11T22:03:10.1614104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpff9p4o_f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1614383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1614509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1614791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1614927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1615113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v0bad_n
2023-01-11T22:03:10.1615310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v0bad_n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1615476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1615629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1615907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpff9p4o_f/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1616184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v0bad_n/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1616467Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpff9p4o_f/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1616536Z ok (2.540s)
2023-01-11T22:03:10.1616542Z 
2023-01-11T22:03:10.1616741Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1616816Z Ran 1 test in 2.540s
2023-01-11T22:03:10.1616822Z 
2023-01-11T22:03:10.1616882Z OK
2023-01-11T22:03:10.1616887Z 
2023-01-11T22:03:10.1616975Z Generating XML reports...
2023-01-11T22:03:10.1617347Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213906.xml
2023-01-11T22:03:10.1617669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1617795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1618078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1618214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1618398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7d1nawy5
2023-01-11T22:03:10.1618595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7d1nawy5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1618851Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1618857Z 
2023-01-11T22:03:10.1618932Z Running tests...
2023-01-11T22:03:10.1619118Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1619362Z   test_forward_async_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1136
2023-01-11T22:03:10.1619519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1137
2023-01-11T22:03:10.1619796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1619923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1620207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1620344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1620531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vdwi93h
2023-01-11T22:03:10.1620731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vdwi93h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1620993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1621118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1621399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1621534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1621847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_vgo9z1
2023-01-11T22:03:10.1622044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_vgo9z1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1622210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1622377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1622654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_vgo9z1/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1622913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vdwi93h/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1623191Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmp6vdwi93h/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1623265Z ok (2.540s)
2023-01-11T22:03:10.1623270Z 
2023-01-11T22:03:10.1623467Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1623543Z Ran 1 test in 2.541s
2023-01-11T22:03:10.1623549Z 
2023-01-11T22:03:10.1623609Z OK
2023-01-11T22:03:10.1623661Z 
2023-01-11T22:03:10.1623747Z Generating XML reports...
2023-01-11T22:03:10.1624153Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213910.xml
2023-01-11T22:03:10.1624434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1624544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1624829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1624965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1625151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo17acmu4
2023-01-11T22:03:10.1625347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo17acmu4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1625603Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1625612Z 
2023-01-11T22:03:10.1625686Z Running tests...
2023-01-11T22:03:10.1625885Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1626117Z   test_forward_sync (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1257
2023-01-11T22:03:10.1626260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1258
2023-01-11T22:03:10.1626536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1626659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1626940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1627079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1627269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkfz_upq
2023-01-11T22:03:10.1627469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkfz_upq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1627740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1627851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1628132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1628268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1628453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbpspqzy
2023-01-11T22:03:10.1628649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbpspqzy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1628815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1628982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1629258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkfz_upq/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1629532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbpspqzy/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1629812Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmprbpspqzy/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1629867Z ok (2.529s)
2023-01-11T22:03:10.1629873Z 
2023-01-11T22:03:10.1630072Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1630181Z Ran 1 test in 2.529s
2023-01-11T22:03:10.1630187Z 
2023-01-11T22:03:10.1630252Z OK
2023-01-11T22:03:10.1630257Z 
2023-01-11T22:03:10.1630342Z Generating XML reports...
2023-01-11T22:03:10.1630739Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213915.xml
2023-01-11T22:03:10.1631018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1631143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1631418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1631556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1631744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppydiqets
2023-01-11T22:03:10.1631942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppydiqets/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1632199Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1632205Z 
2023-01-11T22:03:10.1632283Z Running tests...
2023-01-11T22:03:10.1632484Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1632722Z   test_forward_sync_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1378
2023-01-11T22:03:10.1632878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1379
2023-01-11T22:03:10.1633140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1633264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1633544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1633681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1633869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbsqeuw_i
2023-01-11T22:03:10.1634070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbsqeuw_i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1634342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1634465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1634745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1634868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1635053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0crfp5ez
2023-01-11T22:03:10.1635250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0crfp5ez/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1635418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1635585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1635861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbsqeuw_i/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1636135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0crfp5ez/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1636412Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpbsqeuw_i/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1636480Z ok (2.531s)
2023-01-11T22:03:10.1636486Z 
2023-01-11T22:03:10.1636711Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1636789Z Ran 1 test in 2.531s
2023-01-11T22:03:10.1636794Z 
2023-01-11T22:03:10.1636855Z OK
2023-01-11T22:03:10.1636861Z 
2023-01-11T22:03:10.1636976Z Generating XML reports...
2023-01-11T22:03:10.1637347Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213919.xml
2023-01-11T22:03:10.1637623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1637749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1638031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1638168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1638344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvwcjjkiw
2023-01-11T22:03:10.1638545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvwcjjkiw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1638801Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1638806Z 
2023-01-11T22:03:10.1638956Z Running tests...
2023-01-11T22:03:10.1639159Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1639397Z   test_forward_with_kwargs (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1499
2023-01-11T22:03:10.1639553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1500
2023-01-11T22:03:10.1639829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1639952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1640221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1640361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1640548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3164o1in
2023-01-11T22:03:10.1640743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3164o1in/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1641019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1641145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1641427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1641563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1641734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3c219ubb
2023-01-11T22:03:10.1641931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3c219ubb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1642098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1642263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1642332Z ok (2.442s)
2023-01-11T22:03:10.1642338Z 
2023-01-11T22:03:10.1642534Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1642610Z Ran 1 test in 2.442s
2023-01-11T22:03:10.1642616Z 
2023-01-11T22:03:10.1642675Z OK
2023-01-11T22:03:10.1642680Z 
2023-01-11T22:03:10.1642752Z Generating XML reports...
2023-01-11T22:03:10.1643118Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213924.xml
2023-01-11T22:03:10.1643391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1643550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1643934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1644072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1644258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxargtqxw
2023-01-11T22:03:10.1644457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxargtqxw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1644712Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1644718Z 
2023-01-11T22:03:10.1644779Z Running tests...
2023-01-11T22:03:10.1644975Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1645213Z   test_get_module_rref (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1616
2023-01-11T22:03:10.1645375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1617
2023-01-11T22:03:10.1645653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1645777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1646059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1646194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1646384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvuvfl5od
2023-01-11T22:03:10.1646568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvuvfl5od/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1646839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1646966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1647252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1647389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1647577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvjywhj_
2023-01-11T22:03:10.1647774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvjywhj_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1647939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1648091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1648159Z ok (2.443s)
2023-01-11T22:03:10.1648164Z 
2023-01-11T22:03:10.1648364Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1648440Z Ran 1 test in 2.443s
2023-01-11T22:03:10.1648446Z 
2023-01-11T22:03:10.1648508Z OK
2023-01-11T22:03:10.1648515Z 
2023-01-11T22:03:10.1648601Z Generating XML reports...
2023-01-11T22:03:10.1648970Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213928.xml
2023-01-11T22:03:10.1649245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1649369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1649636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1649772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1649959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmnkcpk89
2023-01-11T22:03:10.1650195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmnkcpk89/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1650479Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1650485Z 
2023-01-11T22:03:10.1650560Z Running tests...
2023-01-11T22:03:10.1650754Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1651015Z   test_remote_module_py_pickle_not_supported (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1733
2023-01-11T22:03:10.1651172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1734
2023-01-11T22:03:10.1651435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1651558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1651843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1651980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1652168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3l6wkgec
2023-01-11T22:03:10.1652363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3l6wkgec/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1652634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1652759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1653026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1653163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1653346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiu69r679
2023-01-11T22:03:10.1653542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiu69r679/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1653709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1653876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1653944Z ok (2.341s)
2023-01-11T22:03:10.1653950Z 
2023-01-11T22:03:10.1654145Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1654218Z Ran 1 test in 2.341s
2023-01-11T22:03:10.1654223Z 
2023-01-11T22:03:10.1654287Z OK
2023-01-11T22:03:10.1654291Z 
2023-01-11T22:03:10.1654376Z Generating XML reports...
2023-01-11T22:03:10.1654733Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213932.xml
2023-01-11T22:03:10.1655013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1655143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1655428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1655565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1655751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ul0pn10
2023-01-11T22:03:10.1655947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ul0pn10/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1656204Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1656209Z 
2023-01-11T22:03:10.1656283Z Running tests...
2023-01-11T22:03:10.1656467Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1656733Z   test_remote_module_py_pickle_not_supported_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1850
2023-01-11T22:03:10.1656966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1851
2023-01-11T22:03:10.1657245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1657370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1657653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1657792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1657981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdef_s0ig
2023-01-11T22:03:10.1658162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdef_s0ig/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1658448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1658573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1658857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1658995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1659182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl4zumybr
2023-01-11T22:03:10.1659377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl4zumybr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1659543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1659708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1659970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdef_s0ig/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1660247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl4zumybr/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1660529Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpl4zumybr/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.1660599Z ok (2.442s)
2023-01-11T22:03:10.1660605Z 
2023-01-11T22:03:10.1660804Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1660881Z Ran 1 test in 2.442s
2023-01-11T22:03:10.1660887Z 
2023-01-11T22:03:10.1660949Z OK
2023-01-11T22:03:10.1660955Z 
2023-01-11T22:03:10.1661043Z Generating XML reports...
2023-01-11T22:03:10.1661410Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213937.xml
2023-01-11T22:03:10.1661799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1661930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1662213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1662351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1662540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbud7ki11
2023-01-11T22:03:10.1662770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbud7ki11/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1663088Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1663094Z 
2023-01-11T22:03:10.1663169Z Running tests...
2023-01-11T22:03:10.1663446Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1663708Z   test_remote_parameters (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1967
2023-01-11T22:03:10.1663868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1968
2023-01-11T22:03:10.1664148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1664272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1664558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1664695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1664881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzooc1ld
2023-01-11T22:03:10.1665076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzooc1ld/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1665355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1665467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1665755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1665893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1666078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89lf6d3k
2023-01-11T22:03:10.1666270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89lf6d3k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1666434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1666599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1666669Z ok (2.445s)
2023-01-11T22:03:10.1666674Z 
2023-01-11T22:03:10.1666857Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1666934Z Ran 1 test in 2.445s
2023-01-11T22:03:10.1666939Z 
2023-01-11T22:03:10.1667004Z OK
2023-01-11T22:03:10.1667009Z 
2023-01-11T22:03:10.1667093Z Generating XML reports...
2023-01-11T22:03:10.1667462Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213941.xml
2023-01-11T22:03:10.1667736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1667859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1668140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1668275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1668448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02_4mrx8
2023-01-11T22:03:10.1668641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02_4mrx8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1668896Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1668902Z 
2023-01-11T22:03:10.1668974Z Running tests...
2023-01-11T22:03:10.1669170Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1669454Z   test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2084
2023-01-11T22:03:10.1669611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2085
2023-01-11T22:03:10.1669884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1670040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1670338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1670474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1670661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdg4zegau
2023-01-11T22:03:10.1670857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdg4zegau/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1671127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1671251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1671532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1671669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1671845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29hn97mp
2023-01-11T22:03:10.1672037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29hn97mp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1672201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1672366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1672693Z The new attribute ``new_attr`` of RemoteModule is ignored during RPC pickling. To pickle this attribute, please add it to ``_REMOTE_MODULE_PICKLED_ATTRIBUTES``. Otherwise, please explicitly add it to ``_REMOTE_MODULE_ATTRIBUTES_IGNORE_FOR_PICKLING``.
2023-01-11T22:03:10.1672761Z ok (2.433s)
2023-01-11T22:03:10.1672767Z 
2023-01-11T22:03:10.1672961Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1673039Z Ran 1 test in 2.433s
2023-01-11T22:03:10.1673045Z 
2023-01-11T22:03:10.1673105Z OK
2023-01-11T22:03:10.1673110Z 
2023-01-11T22:03:10.1673181Z Generating XML reports...
2023-01-11T22:03:10.1673555Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213945.xml
2023-01-11T22:03:10.1673832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1673955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1674238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1674374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1674558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24zumido
2023-01-11T22:03:10.1674757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24zumido/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1675013Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1675019Z 
2023-01-11T22:03:10.1675080Z Running tests...
2023-01-11T22:03:10.1675277Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1675507Z   test_train_eval (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2201
2023-01-11T22:03:10.1675663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2202
2023-01-11T22:03:10.1675939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1676064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1676350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1676519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1676719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3j9e19gs
2023-01-11T22:03:10.1676915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3j9e19gs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1677188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1677315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1677596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1677732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1677916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptxlvtcwp
2023-01-11T22:03:10.1678116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptxlvtcwp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1678282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1678434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1678503Z ok (2.436s)
2023-01-11T22:03:10.1678508Z 
2023-01-11T22:03:10.1678705Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1678781Z Ran 1 test in 2.436s
2023-01-11T22:03:10.1678787Z 
2023-01-11T22:03:10.1678911Z OK
2023-01-11T22:03:10.1678917Z 
2023-01-11T22:03:10.1679004Z Generating XML reports...
2023-01-11T22:03:10.1679376Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213950.xml
2023-01-11T22:03:10.1679652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1679768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1680052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1680191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1680378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphdyz95br
2023-01-11T22:03:10.1680576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphdyz95br/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1680832Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1680838Z 
2023-01-11T22:03:10.1680912Z Running tests...
2023-01-11T22:03:10.1681112Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1681356Z   test_unsupported_methods (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2318
2023-01-11T22:03:10.1681501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2319
2023-01-11T22:03:10.1681778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1681902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1682187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1682323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1682507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5s5jm1uy
2023-01-11T22:03:10.1682700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5s5jm1uy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1682970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1683142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1683439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1683575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1683758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_1vq0b_
2023-01-11T22:03:10.1683951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_1vq0b_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1684116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1684283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1684351Z ok (2.447s)
2023-01-11T22:03:10.1684357Z 
2023-01-11T22:03:10.1684552Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1684617Z Ran 1 test in 2.447s
2023-01-11T22:03:10.1684622Z 
2023-01-11T22:03:10.1684682Z OK
2023-01-11T22:03:10.1684687Z 
2023-01-11T22:03:10.1684770Z Generating XML reports...
2023-01-11T22:03:10.1685141Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213954.xml
2023-01-11T22:03:10.1685414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1685540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1685824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1685961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1686146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7cgztjdk
2023-01-11T22:03:10.1686327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7cgztjdk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1686586Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1686594Z 
2023-01-11T22:03:10.1686669Z Running tests...
2023-01-11T22:03:10.1686868Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1687077Z   test_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2435
2023-01-11T22:03:10.1687233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2436
2023-01-11T22:03:10.1687386Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2437
2023-01-11T22:03:10.1687538Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2438
2023-01-11T22:03:10.1687801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1687927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1688213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1688349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1688534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx6zou5w5
2023-01-11T22:03:10.1688729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx6zou5w5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1689001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1689125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1689406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1689529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1689751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpknfpfijm
2023-01-11T22:03:10.1689980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpknfpfijm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1690258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1690384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1690668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1690803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1690985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgsakyo2w
2023-01-11T22:03:10.1691167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgsakyo2w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1691439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1691564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1691843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1691976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1692157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6crx5otn
2023-01-11T22:03:10.1692351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6crx5otn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1692515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1692681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1692832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1692992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1693061Z ok (2.746s)
2023-01-11T22:03:10.1693066Z 
2023-01-11T22:03:10.1693262Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1693337Z Ran 1 test in 2.747s
2023-01-11T22:03:10.1693342Z 
2023-01-11T22:03:10.1693403Z OK
2023-01-11T22:03:10.1693407Z 
2023-01-11T22:03:10.1693491Z Generating XML reports...
2023-01-11T22:03:10.1693830Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111213958.xml
2023-01-11T22:03:10.1694092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1694214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1694492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1694629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1694815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_w2y69r
2023-01-11T22:03:10.1695007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_w2y69r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1695262Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1695267Z 
2023-01-11T22:03:10.1695340Z Running tests...
2023-01-11T22:03:10.1695536Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1695747Z   test_add_done_callback (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2658
2023-01-11T22:03:10.1695902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2659
2023-01-11T22:03:10.1696092Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2660
2023-01-11T22:03:10.1696278Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2661
2023-01-11T22:03:10.1696554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1696678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1696959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1697096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1697280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1cta13js
2023-01-11T22:03:10.1697463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1cta13js/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1697736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1697864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1698146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1698282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1698466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5owg1zn
2023-01-11T22:03:10.1698661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5owg1zn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1698933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1699044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1699320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1699460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1699645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9q5ylneg
2023-01-11T22:03:10.1699840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9q5ylneg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1700112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1700235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1700513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1700647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1700817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nwyhn1f
2023-01-11T22:03:10.1701012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nwyhn1f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1701179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1701343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1701505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1701777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1701845Z ok (2.751s)
2023-01-11T22:03:10.1701851Z 
2023-01-11T22:03:10.1702051Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1702115Z Ran 1 test in 2.751s
2023-01-11T22:03:10.1702120Z 
2023-01-11T22:03:10.1702183Z OK
2023-01-11T22:03:10.1702188Z 
2023-01-11T22:03:10.1702272Z Generating XML reports...
2023-01-11T22:03:10.1702618Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214003.xml
2023-01-11T22:03:10.1703004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1703131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1703413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1703549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1703736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmw72jr1s
2023-01-11T22:03:10.1703919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmw72jr1s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1704175Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1704181Z 
2023-01-11T22:03:10.1704257Z Running tests...
2023-01-11T22:03:10.1704457Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1704678Z   test_add_with_id (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2881
2023-01-11T22:03:10.1704836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2882
2023-01-11T22:03:10.1704990Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2883
2023-01-11T22:03:10.1705144Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2884
2023-01-11T22:03:10.1705408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1705533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1705818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1705957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1706148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3qgr8pyn
2023-01-11T22:03:10.1706349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3qgr8pyn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1706623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1706749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1707033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1707156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1707342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocvvi6ue
2023-01-11T22:03:10.1707539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocvvi6ue/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1707814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1707945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1708227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1708362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1708549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4lrp110
2023-01-11T22:03:10.1708731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4lrp110/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1709003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1709128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1709445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1709616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1709804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4zejpcz
2023-01-11T22:03:10.1710000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4zejpcz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1710167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1710331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1710481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1710646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1710714Z ok (2.645s)
2023-01-11T22:03:10.1710722Z 
2023-01-11T22:03:10.1710923Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1711000Z Ran 1 test in 2.646s
2023-01-11T22:03:10.1711006Z 
2023-01-11T22:03:10.1711068Z OK
2023-01-11T22:03:10.1711075Z 
2023-01-11T22:03:10.1711160Z Generating XML reports...
2023-01-11T22:03:10.1711500Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214008.xml
2023-01-11T22:03:10.1711760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1711888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1712171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1712307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1712494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmh5t3xx
2023-01-11T22:03:10.1712695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmh5t3xx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1712955Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1712961Z 
2023-01-11T22:03:10.1713036Z Running tests...
2023-01-11T22:03:10.1713234Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1713435Z   test_all_gather (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3104
2023-01-11T22:03:10.1713592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3105
2023-01-11T22:03:10.1713748Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3106
2023-01-11T22:03:10.1713900Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3107
2023-01-11T22:03:10.1714176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1714304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1714591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1714728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1714903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpayrus118
2023-01-11T22:03:10.1715103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpayrus118/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1715382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1715507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1715791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1715959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1716174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7tfo2k6l
2023-01-11T22:03:10.1716371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7tfo2k6l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1716645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1716756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1717037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1717173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1717356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4zh554s8
2023-01-11T22:03:10.1717551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4zh554s8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1717825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1717949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1718230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1718365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1718534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_s8_uws
2023-01-11T22:03:10.1718724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_s8_uws/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1718975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1719146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1719317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1719482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1719552Z ok (2.733s)
2023-01-11T22:03:10.1719558Z 
2023-01-11T22:03:10.1719758Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1719820Z Ran 1 test in 2.734s
2023-01-11T22:03:10.1719825Z 
2023-01-11T22:03:10.1719888Z OK
2023-01-11T22:03:10.1719893Z 
2023-01-11T22:03:10.1719979Z Generating XML reports...
2023-01-11T22:03:10.1720324Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214012.xml
2023-01-11T22:03:10.1720600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1720726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1721010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1721150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1721337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgl3ydx7_
2023-01-11T22:03:10.1721519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgl3ydx7_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1721777Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1721783Z 
2023-01-11T22:03:10.1721857Z Running tests...
2023-01-11T22:03:10.1722054Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1722274Z   test_all_gather_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3327
2023-01-11T22:03:10.1722467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3328
2023-01-11T22:03:10.1722620Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3330
2023-01-11T22:03:10.1722799Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3331
2023-01-11T22:03:10.1723064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1723188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1723468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1723603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1723789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprswy22j3
2023-01-11T22:03:10.1723987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprswy22j3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1724261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1724390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1724670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1724792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1724975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3x7mab87
2023-01-11T22:03:10.1725167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3x7mab87/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1725440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1725564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1725848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1726032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1726217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8oi4nn_o
2023-01-11T22:03:10.1726398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8oi4nn_o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1726668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1726791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1727068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1727207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1727392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm945pyd9
2023-01-11T22:03:10.1727589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm945pyd9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1727757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1727923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1728073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1728235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1728508Z [W tensorpipe_agent.cpp:939] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.1728795Z [W tensorpipe_agent.cpp:624] RPC agent for worker3 won't send response to request #5 to worker0, as the agent is shutting down
2023-01-11T22:03:10.1728900Z ok (11.566s)
2023-01-11T22:03:10.1728906Z 
2023-01-11T22:03:10.1729105Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1729208Z Ran 1 test in 11.567s
2023-01-11T22:03:10.1729215Z 
2023-01-11T22:03:10.1729275Z OK
2023-01-11T22:03:10.1729280Z 
2023-01-11T22:03:10.1729363Z Generating XML reports...
2023-01-11T22:03:10.1729689Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214017.xml
2023-01-11T22:03:10.1729964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1730088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1730370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1730506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1730694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9hr1jw3
2023-01-11T22:03:10.1730890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9hr1jw3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1731146Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1731151Z 
2023-01-11T22:03:10.1731223Z Running tests...
2023-01-11T22:03:10.1731408Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1731620Z   test_async_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3551
2023-01-11T22:03:10.1731773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3552
2023-01-11T22:03:10.1731926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3553
2023-01-11T22:03:10.1732076Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3554
2023-01-11T22:03:10.1732353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1732479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1732760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1732883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1733069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5qvaj3u0
2023-01-11T22:03:10.1733263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5qvaj3u0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1733533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1733657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1733939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1734076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1734260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yenmyg8
2023-01-11T22:03:10.1734443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yenmyg8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1734713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1734835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1735116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1735251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1735436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphrkmng_9
2023-01-11T22:03:10.1735666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphrkmng_9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1735964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1736088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1736356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1736491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1736674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphn6rlcwn
2023-01-11T22:03:10.1736870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphn6rlcwn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1737036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1737203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1737367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1737529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1737583Z ok (2.644s)
2023-01-11T22:03:10.1737600Z 
2023-01-11T22:03:10.1737785Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1737860Z Ran 1 test in 2.644s
2023-01-11T22:03:10.1737867Z 
2023-01-11T22:03:10.1737927Z OK
2023-01-11T22:03:10.1737933Z 
2023-01-11T22:03:10.1738016Z Generating XML reports...
2023-01-11T22:03:10.1738358Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214031.xml
2023-01-11T22:03:10.1738630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1738757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1739043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1739167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1739356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpotk29pzk
2023-01-11T22:03:10.1739551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpotk29pzk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1739804Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1739809Z 
2023-01-11T22:03:10.1739882Z Running tests...
2023-01-11T22:03:10.1740075Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1740297Z   test_async_class_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3774
2023-01-11T22:03:10.1740457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3775
2023-01-11T22:03:10.1740612Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3776
2023-01-11T22:03:10.1740751Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3777
2023-01-11T22:03:10.1741025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1741147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1741427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1741562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1741867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8n3n82_z
2023-01-11T22:03:10.1742118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8n3n82_z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1742428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1742541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1742823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1742957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1743144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpudgy7e3z
2023-01-11T22:03:10.1743340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpudgy7e3z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1743615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1743744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1744029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1744163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1744334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkm0ar84
2023-01-11T22:03:10.1744529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkm0ar84/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1744805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1744928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1745210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1745347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1745534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyf11zp0
2023-01-11T22:03:10.1745732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyf11zp0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1745883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1746050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1746210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1746372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1746439Z ok (2.548s)
2023-01-11T22:03:10.1746448Z 
2023-01-11T22:03:10.1746647Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1746724Z Ran 1 test in 2.548s
2023-01-11T22:03:10.1746732Z 
2023-01-11T22:03:10.1746792Z OK
2023-01-11T22:03:10.1746797Z 
2023-01-11T22:03:10.1746879Z Generating XML reports...
2023-01-11T22:03:10.1747210Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214035.xml
2023-01-11T22:03:10.1747488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1747613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1747894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1748029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1748214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexln7q94
2023-01-11T22:03:10.1748409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexln7q94/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1748741Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1748747Z 
2023-01-11T22:03:10.1748820Z Running tests...
2023-01-11T22:03:10.1749044Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1749277Z   test_async_class_method_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3997
2023-01-11T22:03:10.1749435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3998
2023-01-11T22:03:10.1749589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3999
2023-01-11T22:03:10.1749740Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4000
2023-01-11T22:03:10.1750015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1750141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1750427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1750551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1750741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnc8cwir
2023-01-11T22:03:10.1750939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnc8cwir/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1751216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1751340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1751620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1751756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1751945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4h4dzy3
2023-01-11T22:03:10.1752217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1752327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1752606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1752741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1752927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplimu0nzg
2023-01-11T22:03:10.1753123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4h4dzy3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1753316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplimu0nzg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1753590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1753716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1753986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1754124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1754310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqj8ta36r
2023-01-11T22:03:10.1754502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqj8ta36r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1754666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1754831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1754993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1755199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1755254Z ok (2.649s)
2023-01-11T22:03:10.1755272Z 
2023-01-11T22:03:10.1755482Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1755559Z Ran 1 test in 2.650s
2023-01-11T22:03:10.1755564Z 
2023-01-11T22:03:10.1755623Z OK
2023-01-11T22:03:10.1755629Z 
2023-01-11T22:03:10.1755712Z Generating XML reports...
2023-01-11T22:03:10.1756047Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214040.xml
2023-01-11T22:03:10.1756322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1756447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1756728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1756854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1757041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpede9fpx3
2023-01-11T22:03:10.1757239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpede9fpx3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1757495Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1757500Z 
2023-01-11T22:03:10.1757573Z Running tests...
2023-01-11T22:03:10.1757769Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1757994Z   test_async_class_rref_proxy (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4220
2023-01-11T22:03:10.1758148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4221
2023-01-11T22:03:10.1758300Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4222
2023-01-11T22:03:10.1758443Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4223
2023-01-11T22:03:10.1758720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1758933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1759223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1759361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1759549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvygsdd9
2023-01-11T22:03:10.1759747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvygsdd9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1760023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1760138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1760423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1760560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1760746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38dfawxe
2023-01-11T22:03:10.1760944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38dfawxe/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1761221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1761346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1761630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1761806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1761980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmz5tbwc
2023-01-11T22:03:10.1762203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmz5tbwc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1762479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1762605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1762887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1763023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1763209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9w90t9y
2023-01-11T22:03:10.1763404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9w90t9y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1763558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1763727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1763892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1764055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1764126Z ok (2.740s)
2023-01-11T22:03:10.1764132Z 
2023-01-11T22:03:10.1764330Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1764406Z Ran 1 test in 2.740s
2023-01-11T22:03:10.1764413Z 
2023-01-11T22:03:10.1764474Z OK
2023-01-11T22:03:10.1764479Z 
2023-01-11T22:03:10.1764563Z Generating XML reports...
2023-01-11T22:03:10.1764887Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214044.xml
2023-01-11T22:03:10.1765165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1765292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1765575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1765712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1765898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkddottln
2023-01-11T22:03:10.1766094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkddottln/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1766350Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1766355Z 
2023-01-11T22:03:10.1766429Z Running tests...
2023-01-11T22:03:10.1766612Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1766843Z   test_async_class_rref_proxy_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4443
2023-01-11T22:03:10.1767002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4444
2023-01-11T22:03:10.1767160Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4445
2023-01-11T22:03:10.1767315Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4446
2023-01-11T22:03:10.1767597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1767723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1768005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1768127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1768351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvpml9mt
2023-01-11T22:03:10.1768576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvpml9mt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1768853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1768978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1769259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1769397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1769583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzqomthr
2023-01-11T22:03:10.1769766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzqomthr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1770043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1770167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1770455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1770590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1770776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpypaaw83r
2023-01-11T22:03:10.1770972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpypaaw83r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1771249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1771373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1771640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1771776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1771962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi4noxihs
2023-01-11T22:03:10.1772155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi4noxihs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1772319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1772484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1772647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1772810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1772866Z ok (2.741s)
2023-01-11T22:03:10.1772883Z 
2023-01-11T22:03:10.1773065Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1773142Z Ran 1 test in 2.741s
2023-01-11T22:03:10.1773147Z 
2023-01-11T22:03:10.1773208Z OK
2023-01-11T22:03:10.1773214Z 
2023-01-11T22:03:10.1773299Z Generating XML reports...
2023-01-11T22:03:10.1773640Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214049.xml
2023-01-11T22:03:10.1773911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1774035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1774316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1774439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1774623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppso1m7ds
2023-01-11T22:03:10.1774854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppso1m7ds/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1775138Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1775145Z 
2023-01-11T22:03:10.1775218Z Running tests...
2023-01-11T22:03:10.1775416Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1775647Z   test_async_class_rref_proxy_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4666
2023-01-11T22:03:10.1775802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4667
2023-01-11T22:03:10.1775955Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4668
2023-01-11T22:03:10.1776096Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4669
2023-01-11T22:03:10.1776367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1776493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1776775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1776910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1777097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wbhnkzt
2023-01-11T22:03:10.1777295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wbhnkzt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1777568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1777680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1777961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1778097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1778284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_ulape4
2023-01-11T22:03:10.1778479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_ulape4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1778751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1778875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1779153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1779288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1779460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyv96ukkh
2023-01-11T22:03:10.1779654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyv96ukkh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1779930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1780053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1780331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1780466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1780650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnsrlzby
2023-01-11T22:03:10.1780845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnsrlzby/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1780997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1781164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1781365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1781556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1781626Z ok (2.747s)
2023-01-11T22:03:10.1781632Z 
2023-01-11T22:03:10.1781948Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1782026Z Ran 1 test in 2.747s
2023-01-11T22:03:10.1782032Z 
2023-01-11T22:03:10.1782094Z OK
2023-01-11T22:03:10.1782099Z 
2023-01-11T22:03:10.1782183Z Generating XML reports...
2023-01-11T22:03:10.1782512Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214054.xml
2023-01-11T22:03:10.1782788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1782920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1783206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1783354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1783621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnr9izhlb
2023-01-11T22:03:10.1783845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnr9izhlb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1784104Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1784110Z 
2023-01-11T22:03:10.1784182Z Running tests...
2023-01-11T22:03:10.1784368Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1784594Z   test_async_function_chained (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4889
2023-01-11T22:03:10.1784753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4890
2023-01-11T22:03:10.1784907Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4891
2023-01-11T22:03:10.1785064Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4892
2023-01-11T22:03:10.1785339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1785463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1785747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1785871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1786055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gow5zmh
2023-01-11T22:03:10.1786250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gow5zmh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1786524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1786649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1786929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1787067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1787251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpev2_i885
2023-01-11T22:03:10.1787443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpev2_i885/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1787703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1787827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1788110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1788316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1788541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprk9fe0_i
2023-01-11T22:03:10.1788739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprk9fe0_i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1789014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1789138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1789409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1789545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1789730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppt8xmi71
2023-01-11T22:03:10.1789931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppt8xmi71/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1790098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1790263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1790428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1790590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1790648Z ok (2.750s)
2023-01-11T22:03:10.1790669Z 
2023-01-11T22:03:10.1790854Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1790929Z Ran 1 test in 2.750s
2023-01-11T22:03:10.1790935Z 
2023-01-11T22:03:10.1790995Z OK
2023-01-11T22:03:10.1791000Z 
2023-01-11T22:03:10.1791084Z Generating XML reports...
2023-01-11T22:03:10.1791426Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214058.xml
2023-01-11T22:03:10.1791703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1791827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1792107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1792230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1792414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxg2ujeuz
2023-01-11T22:03:10.1792609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxg2ujeuz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1792866Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1792874Z 
2023-01-11T22:03:10.1792949Z Running tests...
2023-01-11T22:03:10.1793145Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1793381Z   test_async_function_chained_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5112
2023-01-11T22:03:10.1793537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5113
2023-01-11T22:03:10.1793691Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5114
2023-01-11T22:03:10.1793830Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5115
2023-01-11T22:03:10.1794114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1794301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1794701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1794905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1795142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbjt05j7
2023-01-11T22:03:10.1795342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbjt05j7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1795617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1795728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1796011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1796147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1796333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11p_6bie
2023-01-11T22:03:10.1796532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11p_6bie/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1796805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1796932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1797210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1797345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1797517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ohxqjiv
2023-01-11T22:03:10.1797711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ohxqjiv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1797983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1798112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1798391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1798529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1798713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7vla6al
2023-01-11T22:03:10.1798977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7vla6al/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1799130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1799294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1799458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1799623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1799695Z ok (2.649s)
2023-01-11T22:03:10.1799702Z 
2023-01-11T22:03:10.1799901Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1799980Z Ran 1 test in 2.649s
2023-01-11T22:03:10.1799985Z 
2023-01-11T22:03:10.1800046Z OK
2023-01-11T22:03:10.1800051Z 
2023-01-11T22:03:10.1800136Z Generating XML reports...
2023-01-11T22:03:10.1800460Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214103.xml
2023-01-11T22:03:10.1800736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1800861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1801140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1801277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1801499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg3gy0007
2023-01-11T22:03:10.1801723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg3gy0007/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1801979Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1801985Z 
2023-01-11T22:03:10.1802057Z Running tests...
2023-01-11T22:03:10.1802239Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1802470Z   test_async_function_multi_chained (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5335
2023-01-11T22:03:10.1802627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5336
2023-01-11T22:03:10.1802781Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5337
2023-01-11T22:03:10.1802934Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5338
2023-01-11T22:03:10.1803210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1803334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1803616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1803738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1803921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcu0_5t4j
2023-01-11T22:03:10.1804113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcu0_5t4j/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1804385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1804508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1804790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1804926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1805111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kn1o99g
2023-01-11T22:03:10.1805379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1805490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1805770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1805903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1806085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputfhp_5c
2023-01-11T22:03:10.1806355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1806480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1806758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1806891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1807061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_1d6nfk
2023-01-11T22:03:10.1807253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kn1o99g/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1807443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputfhp_5c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1807625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_1d6nfk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1807790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1807984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1808173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1808337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1808405Z ok (2.651s)
2023-01-11T22:03:10.1808411Z 
2023-01-11T22:03:10.1808594Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1808669Z Ran 1 test in 2.652s
2023-01-11T22:03:10.1808674Z 
2023-01-11T22:03:10.1808735Z OK
2023-01-11T22:03:10.1808741Z 
2023-01-11T22:03:10.1808824Z Generating XML reports...
2023-01-11T22:03:10.1809163Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214108.xml
2023-01-11T22:03:10.1809435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1809560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1809840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1809965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1810151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc42fmvtq
2023-01-11T22:03:10.1810347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc42fmvtq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1810603Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1810609Z 
2023-01-11T22:03:10.1810682Z Running tests...
2023-01-11T22:03:10.1810882Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1811119Z   test_async_function_multi_chained_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5558
2023-01-11T22:03:10.1811277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5559
2023-01-11T22:03:10.1811434Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5560
2023-01-11T22:03:10.1811577Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5561
2023-01-11T22:03:10.1811853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1811976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1812260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1812395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1812584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcqsduj7x
2023-01-11T22:03:10.1812782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcqsduj7x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1813057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1813169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1813449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1813584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1813768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3k9w8mgx
2023-01-11T22:03:10.1813962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3k9w8mgx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1814238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1814393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1814707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1814843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1815015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tkvyftf
2023-01-11T22:03:10.1815209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tkvyftf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1815481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1815606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1815886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1816024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1816210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl2oj8lgy
2023-01-11T22:03:10.1816406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl2oj8lgy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1816559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1816728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1816893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1817057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1817126Z ok (2.648s)
2023-01-11T22:03:10.1817133Z 
2023-01-11T22:03:10.1817332Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1817409Z Ran 1 test in 2.648s
2023-01-11T22:03:10.1817416Z 
2023-01-11T22:03:10.1817479Z OK
2023-01-11T22:03:10.1817484Z 
2023-01-11T22:03:10.1817572Z Generating XML reports...
2023-01-11T22:03:10.1817903Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214112.xml
2023-01-11T22:03:10.1818180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1818307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1818589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1818725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1818913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx79kux6s
2023-01-11T22:03:10.1819110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx79kux6s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1819370Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1819376Z 
2023-01-11T22:03:10.1819450Z Running tests...
2023-01-11T22:03:10.1819634Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1819877Z   test_async_function_multi_chained_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5781
2023-01-11T22:03:10.1820035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5782
2023-01-11T22:03:10.1820188Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5783
2023-01-11T22:03:10.1820345Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5784
2023-01-11T22:03:10.1820619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1820746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1821063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1821224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1821415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0r0m0yw
2023-01-11T22:03:10.1821610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0r0m0yw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1822007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1822134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1822419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1822555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1822744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdun1ntsh
2023-01-11T22:03:10.1823020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1823131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1823412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1823548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1823734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4smas18p
2023-01-11T22:03:10.1823930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdun1ntsh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1824123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4smas18p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1824399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1824524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1824793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1824931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1825115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6zhr_4v
2023-01-11T22:03:10.1825307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6zhr_4v/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1825472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1825638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1825802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1825967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1826034Z ok (2.646s)
2023-01-11T22:03:10.1826042Z 
2023-01-11T22:03:10.1826229Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1826308Z Ran 1 test in 2.647s
2023-01-11T22:03:10.1826313Z 
2023-01-11T22:03:10.1826376Z OK
2023-01-11T22:03:10.1826382Z 
2023-01-11T22:03:10.1826465Z Generating XML reports...
2023-01-11T22:03:10.1826803Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214117.xml
2023-01-11T22:03:10.1827076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1827201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1827482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1827663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1827884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9vqyek7
2023-01-11T22:03:10.1828081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9vqyek7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1828340Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1828346Z 
2023-01-11T22:03:10.1828421Z Running tests...
2023-01-11T22:03:10.1828618Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1828849Z   test_async_function_multi_fanout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6004
2023-01-11T22:03:10.1829007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6005
2023-01-11T22:03:10.1829163Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6006
2023-01-11T22:03:10.1829305Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6007
2023-01-11T22:03:10.1829582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1829708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1829989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1830125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1830310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vky7a0a
2023-01-11T22:03:10.1830504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vky7a0a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1830777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1830892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1831174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1831308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1831492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpucsg1xra
2023-01-11T22:03:10.1831688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpucsg1xra/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1831963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1832088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1832372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1832511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1832686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51d8hmyh
2023-01-11T22:03:10.1832879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51d8hmyh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1833153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1833279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1833560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1833694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1833878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevnxsmby
2023-01-11T22:03:10.1834074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevnxsmby/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1834281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1834461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1834626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1834788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1834856Z ok (2.752s)
2023-01-11T22:03:10.1834862Z 
2023-01-11T22:03:10.1835060Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1835136Z Ran 1 test in 2.752s
2023-01-11T22:03:10.1835142Z 
2023-01-11T22:03:10.1835203Z OK
2023-01-11T22:03:10.1835208Z 
2023-01-11T22:03:10.1835291Z Generating XML reports...
2023-01-11T22:03:10.1835619Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214121.xml
2023-01-11T22:03:10.1835893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1836019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1836298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1836433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1836617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfl0m9yv
2023-01-11T22:03:10.1836811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfl0m9yv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1837071Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1837076Z 
2023-01-11T22:03:10.1837150Z Running tests...
2023-01-11T22:03:10.1837336Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1837571Z   test_async_function_multi_fanout_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6227
2023-01-11T22:03:10.1837726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6228
2023-01-11T22:03:10.1837880Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6229
2023-01-11T22:03:10.1838033Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6230
2023-01-11T22:03:10.1838307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1838429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1838710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1838920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1839111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmwbezfe
2023-01-11T22:03:10.1839310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmwbezfe/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1839585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1839709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1839990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1840126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1840313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy73gix7o
2023-01-11T22:03:10.1840512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy73gix7o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1840814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1840981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1841262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1841395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1841580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7l3iu7p
2023-01-11T22:03:10.1841774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7l3iu7p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1842046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1842169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1842440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1842574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1842760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdjz9itgv
2023-01-11T22:03:10.1842957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdjz9itgv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1843123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1843287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1843451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1843610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1843679Z ok (2.745s)
2023-01-11T22:03:10.1843685Z 
2023-01-11T22:03:10.1843870Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1843947Z Ran 1 test in 2.745s
2023-01-11T22:03:10.1843953Z 
2023-01-11T22:03:10.1844015Z OK
2023-01-11T22:03:10.1844022Z 
2023-01-11T22:03:10.1844108Z Generating XML reports...
2023-01-11T22:03:10.1844450Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214126.xml
2023-01-11T22:03:10.1844728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1844854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1845136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1845259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1845444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiordoa7i
2023-01-11T22:03:10.1845643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiordoa7i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1845899Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1845905Z 
2023-01-11T22:03:10.1845978Z Running tests...
2023-01-11T22:03:10.1846176Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1846412Z   test_async_function_multi_fanout_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6450
2023-01-11T22:03:10.1846566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6451
2023-01-11T22:03:10.1846719Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6452
2023-01-11T22:03:10.1846857Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6453
2023-01-11T22:03:10.1847131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1847286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1847633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1847770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1847957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp223mrkcz
2023-01-11T22:03:10.1848154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp223mrkcz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1848427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1848552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1848819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1848958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1849146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl4uut3a5
2023-01-11T22:03:10.1849342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl4uut3a5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1849614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1849740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1850020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1850154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1873227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvp808q2q
2023-01-11T22:03:10.1873557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvp808q2q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1873894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1874015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1874296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1874426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1874607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahy_7bco
2023-01-11T22:03:10.1874798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahy_7bco/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1874956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1875113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1875268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1875423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1875483Z ok (2.745s)
2023-01-11T22:03:10.1875491Z 
2023-01-11T22:03:10.1875685Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1875753Z Ran 1 test in 2.745s
2023-01-11T22:03:10.1875759Z 
2023-01-11T22:03:10.1875812Z OK
2023-01-11T22:03:10.1875817Z 
2023-01-11T22:03:10.1875893Z Generating XML reports...
2023-01-11T22:03:10.1876229Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214131.xml
2023-01-11T22:03:10.1876495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1876772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1877051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1877232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1877415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyld1l8et
2023-01-11T22:03:10.1877604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyld1l8et/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1877855Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1877861Z 
2023-01-11T22:03:10.1877926Z Running tests...
2023-01-11T22:03:10.1878118Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1878334Z   test_async_function_nested (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6673
2023-01-11T22:03:10.1878485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6674
2023-01-11T22:03:10.1878632Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6675
2023-01-11T22:03:10.1878777Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6676
2023-01-11T22:03:10.1879143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1879261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1879538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1879666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1879841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoc69rg30
2023-01-11T22:03:10.1880108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1880226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1880501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1880630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1880808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrps2xf3
2023-01-11T22:03:10.1880995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoc69rg30/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1881179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrps2xf3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1881445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1881557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1881832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1881962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1882141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgghubrcx
2023-01-11T22:03:10.1882331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgghubrcx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1882596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1882711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1882983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1883107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1883332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu70f0iv7
2023-01-11T22:03:10.1883518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu70f0iv7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1883705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1883864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1884020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1884174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1884233Z ok (2.748s)
2023-01-11T22:03:10.1884240Z 
2023-01-11T22:03:10.1884431Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1884495Z Ran 1 test in 2.748s
2023-01-11T22:03:10.1884502Z 
2023-01-11T22:03:10.1884554Z OK
2023-01-11T22:03:10.1884559Z 
2023-01-11T22:03:10.1884636Z Generating XML reports...
2023-01-11T22:03:10.1884968Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214135.xml
2023-01-11T22:03:10.1885241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1885357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1885632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1885758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1885934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpts68alxz
2023-01-11T22:03:10.1886117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpts68alxz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1886364Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1886372Z 
2023-01-11T22:03:10.1886438Z Running tests...
2023-01-11T22:03:10.1886629Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1886857Z   test_async_function_nested_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6896
2023-01-11T22:03:10.1887004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6897
2023-01-11T22:03:10.1887147Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6898
2023-01-11T22:03:10.1887292Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6899
2023-01-11T22:03:10.1887557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1887673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1887948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1888078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1888257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdf3v4ndt
2023-01-11T22:03:10.1888445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdf3v4ndt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1888708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1888823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1889094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1889216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1889391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46w8d74p
2023-01-11T22:03:10.1889610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46w8d74p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1889903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1890021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1890292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1890419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1890594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0oxq2_wl
2023-01-11T22:03:10.1890774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0oxq2_wl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1891037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1891157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1891430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1891558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1891735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgilwwq6x
2023-01-11T22:03:10.1891921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgilwwq6x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1892076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1892232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1892381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1892535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1892598Z ok (2.746s)
2023-01-11T22:03:10.1892604Z 
2023-01-11T22:03:10.1892795Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1892863Z Ran 1 test in 2.747s
2023-01-11T22:03:10.1892869Z 
2023-01-11T22:03:10.1892920Z OK
2023-01-11T22:03:10.1892925Z 
2023-01-11T22:03:10.1893000Z Generating XML reports...
2023-01-11T22:03:10.1893332Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214140.xml
2023-01-11T22:03:10.1893593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1893709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1893981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1894109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1894286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bimxmgv
2023-01-11T22:03:10.1894475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bimxmgv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1894725Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1894730Z 
2023-01-11T22:03:10.1894798Z Running tests...
2023-01-11T22:03:10.1894987Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1895199Z   test_async_function_raise (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7119
2023-01-11T22:03:10.1895346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7120
2023-01-11T22:03:10.1895493Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7121
2023-01-11T22:03:10.1895672Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7122
2023-01-11T22:03:10.1895967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1896084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1896359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1896486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1896660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvdne5rpp
2023-01-11T22:03:10.1896850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvdne5rpp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1897110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1897225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1897500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1897627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1897804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6sjrbpb
2023-01-11T22:03:10.1897992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6sjrbpb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1898253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1898363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1898633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1898760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1898938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7q2qnsei
2023-01-11T22:03:10.1899125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7q2qnsei/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1899388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1899502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1899771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1899894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1900065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdn9lpbv
2023-01-11T22:03:10.1900249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdn9lpbv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1900406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1900565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1900722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1900876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1900959Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.1901075Z RuntimeError('Expected error')
2023-01-11T22:03:10.1901156Z Traceback (most recent call last):
2023-01-11T22:03:10.1901410Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1901536Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1901916Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2023-01-11T22:03:10.1902044Z     return fn(*args, **kwargs)
2023-01-11T22:03:10.1902333Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2023-01-11T22:03:10.1902463Z     raise RuntimeError("Expected error")
2023-01-11T22:03:10.1902542Z RuntimeError: Expected error
2023-01-11T22:03:10.1902553Z 
2023-01-11T22:03:10.1902631Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.1902748Z RuntimeError('Expected error')
2023-01-11T22:03:10.1902833Z Traceback (most recent call last):
2023-01-11T22:03:10.1903085Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1903213Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1903459Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2023-01-11T22:03:10.1903533Z     return fn(*args, **kwargs)
2023-01-11T22:03:10.1903810Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2023-01-11T22:03:10.1903905Z     raise RuntimeError("Expected error")
2023-01-11T22:03:10.1903989Z RuntimeError: Expected error
2023-01-11T22:03:10.1903994Z 
2023-01-11T22:03:10.1904078Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.1904194Z RuntimeError('Expected error')
2023-01-11T22:03:10.1904278Z Traceback (most recent call last):
2023-01-11T22:03:10.1904526Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1904652Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1904892Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2023-01-11T22:03:10.1904966Z     return fn(*args, **kwargs)
2023-01-11T22:03:10.1905250Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2023-01-11T22:03:10.1905354Z     raise RuntimeError("Expected error")
2023-01-11T22:03:10.1905442Z RuntimeError: Expected error
2023-01-11T22:03:10.1905447Z 
2023-01-11T22:03:10.1905541Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.1905663Z RuntimeError('Expected error')
2023-01-11T22:03:10.1905754Z Traceback (most recent call last):
2023-01-11T22:03:10.1905995Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1906127Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1906376Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2023-01-11T22:03:10.1906459Z     return fn(*args, **kwargs)
2023-01-11T22:03:10.1906745Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2023-01-11T22:03:10.1906845Z     raise RuntimeError("Expected error")
2023-01-11T22:03:10.1906935Z RuntimeError: Expected error
2023-01-11T22:03:10.1906940Z 
2023-01-11T22:03:10.1907006Z ok (2.646s)
2023-01-11T22:03:10.1907012Z 
2023-01-11T22:03:10.1907200Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1907275Z Ran 1 test in 2.646s
2023-01-11T22:03:10.1907281Z 
2023-01-11T22:03:10.1907340Z OK
2023-01-11T22:03:10.1907345Z 
2023-01-11T22:03:10.1907430Z Generating XML reports...
2023-01-11T22:03:10.1907779Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214145.xml
2023-01-11T22:03:10.1908053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1908179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1908458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1908628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1908801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp477xny8d
2023-01-11T22:03:10.1909043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp477xny8d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1909302Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1909308Z 
2023-01-11T22:03:10.1909381Z Running tests...
2023-01-11T22:03:10.1909579Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1910278Z   test_async_function_raise_async (__main__.TensorPipeRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76907 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.318s)
2023-01-11T22:03:10.1910288Z 
2023-01-11T22:03:10.1910483Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1910559Z Ran 1 test in 0.318s
2023-01-11T22:03:10.1910564Z 
2023-01-11T22:03:10.1910638Z OK (skipped=1)
2023-01-11T22:03:10.1910644Z 
2023-01-11T22:03:10.1910716Z Generating XML reports...
2023-01-11T22:03:10.1911053Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214149.xml
2023-01-11T22:03:10.1911327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1911453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1911736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1911872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1912061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8361imgc
2023-01-11T22:03:10.1912259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8361imgc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1912514Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1912521Z 
2023-01-11T22:03:10.1912581Z Running tests...
2023-01-11T22:03:10.1912777Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1913006Z   test_async_function_raise_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7352
2023-01-11T22:03:10.1913163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7353
2023-01-11T22:03:10.1913317Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7354
2023-01-11T22:03:10.1913468Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7355
2023-01-11T22:03:10.1913749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1913875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1914159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1914282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1914468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51on_e6_
2023-01-11T22:03:10.1914661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51on_e6_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1914936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1915060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1915340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1915510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1915725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnd4ppoux
2023-01-11T22:03:10.1915911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnd4ppoux/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1916183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1916308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1916592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1916727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1916913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7yurdkzy
2023-01-11T22:03:10.1917112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7yurdkzy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1917387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1917511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1917779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1917915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1918099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5dcfuqu
2023-01-11T22:03:10.1918295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5dcfuqu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1918460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1918627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1918794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1919035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1919116Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.1919245Z RuntimeError('Expected error')
2023-01-11T22:03:10.1919340Z Traceback (most recent call last):
2023-01-11T22:03:10.1919603Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1919739Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1919999Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2023-01-11T22:03:10.1920088Z     return fn(*args, **kwargs)
2023-01-11T22:03:10.1920385Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2023-01-11T22:03:10.1920476Z     raise RuntimeError("Expected error")
2023-01-11T22:03:10.1920569Z RuntimeError: Expected error
2023-01-11T22:03:10.1920574Z 
2023-01-11T22:03:10.1920665Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.1920790Z RuntimeError('Expected error')
2023-01-11T22:03:10.1920881Z Traceback (most recent call last):
2023-01-11T22:03:10.1921139Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1921272Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1921523Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2023-01-11T22:03:10.1921592Z     return fn(*args, **kwargs)
2023-01-11T22:03:10.1921882Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2023-01-11T22:03:10.1922023Z     raise RuntimeError("Expected error")
2023-01-11T22:03:10.1922113Z RuntimeError: Expected error
2023-01-11T22:03:10.1922119Z 
2023-01-11T22:03:10.1922234Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.1922360Z RuntimeError('Expected error')
2023-01-11T22:03:10.1922450Z Traceback (most recent call last):
2023-01-11T22:03:10.1922692Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1922826Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1923077Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2023-01-11T22:03:10.1923160Z     return fn(*args, **kwargs)
2023-01-11T22:03:10.1923450Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2023-01-11T22:03:10.1923551Z     raise RuntimeError("Expected error")
2023-01-11T22:03:10.1923642Z RuntimeError: Expected error
2023-01-11T22:03:10.1923648Z 
2023-01-11T22:03:10.1923739Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.1923850Z RuntimeError('Expected error')
2023-01-11T22:03:10.1923941Z Traceback (most recent call last):
2023-01-11T22:03:10.1924194Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.1924326Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.1924577Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2023-01-11T22:03:10.1924659Z     return fn(*args, **kwargs)
2023-01-11T22:03:10.1924947Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2023-01-11T22:03:10.1925047Z     raise RuntimeError("Expected error")
2023-01-11T22:03:10.1925124Z RuntimeError: Expected error
2023-01-11T22:03:10.1925145Z 
2023-01-11T22:03:10.1925200Z ok (2.648s)
2023-01-11T22:03:10.1925205Z 
2023-01-11T22:03:10.1925404Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1925482Z Ran 1 test in 2.649s
2023-01-11T22:03:10.1925488Z 
2023-01-11T22:03:10.1925549Z OK
2023-01-11T22:03:10.1925554Z 
2023-01-11T22:03:10.1925637Z Generating XML reports...
2023-01-11T22:03:10.1925980Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214152.xml
2023-01-11T22:03:10.1926253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1926379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1926647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1926783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1926972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fx2ahj7
2023-01-11T22:03:10.1927170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fx2ahj7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1927425Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1927431Z 
2023-01-11T22:03:10.1927504Z Running tests...
2023-01-11T22:03:10.1927703Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1927928Z   test_async_function_simple (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7575
2023-01-11T22:03:10.1928072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7576
2023-01-11T22:03:10.1928226Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7577
2023-01-11T22:03:10.1928378Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7578
2023-01-11T22:03:10.1928688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1928838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1929122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1929257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1929447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprg98a1t_
2023-01-11T22:03:10.1929641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprg98a1t_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1929900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1930024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1930306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1930442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1930627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9n7u7g8
2023-01-11T22:03:10.1930898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1931021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1931300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1931421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1931605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lihupz5
2023-01-11T22:03:10.1931877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1932003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1932286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1932420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1932603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7goixg93
2023-01-11T22:03:10.1932798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9n7u7g8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1932989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lihupz5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1933164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7goixg93/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1933329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1933497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1933660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1933825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1933894Z ok (2.650s)
2023-01-11T22:03:10.1933901Z 
2023-01-11T22:03:10.1934098Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1934172Z Ran 1 test in 2.650s
2023-01-11T22:03:10.1934178Z 
2023-01-11T22:03:10.1934227Z OK
2023-01-11T22:03:10.1934246Z 
2023-01-11T22:03:10.1934317Z Generating XML reports...
2023-01-11T22:03:10.1934655Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214156.xml
2023-01-11T22:03:10.1934932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1935090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1935402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1935540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1935726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptb1lkutg
2023-01-11T22:03:10.1935921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptb1lkutg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1936166Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1936184Z 
2023-01-11T22:03:10.1936245Z Running tests...
2023-01-11T22:03:10.1936440Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1936675Z   test_async_function_with_future_ctor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7798
2023-01-11T22:03:10.1936834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7799
2023-01-11T22:03:10.1936989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7800
2023-01-11T22:03:10.1937143Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7801
2023-01-11T22:03:10.1937417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1937541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1937809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1937945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1938129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfu_0tnu
2023-01-11T22:03:10.1938327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfu_0tnu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1938603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1938727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1939006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1939141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1939312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8nj21ps3
2023-01-11T22:03:10.1939506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8nj21ps3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1939778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1939904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1940185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1940319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1940503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy3r6rckk
2023-01-11T22:03:10.1940696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy3r6rckk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1940966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1941076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1941356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1941489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1941853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0ki9zh0
2023-01-11T22:03:10.1942167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0ki9zh0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1942336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1942503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1942666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1942815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1942885Z ok (2.643s)
2023-01-11T22:03:10.1942892Z 
2023-01-11T22:03:10.1943097Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1943173Z Ran 1 test in 2.643s
2023-01-11T22:03:10.1943180Z 
2023-01-11T22:03:10.1943243Z OK
2023-01-11T22:03:10.1943247Z 
2023-01-11T22:03:10.1943334Z Generating XML reports...
2023-01-11T22:03:10.1943681Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214201.xml
2023-01-11T22:03:10.1943959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1944084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1944351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1944487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1944671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqeh1m9n
2023-01-11T22:03:10.1944867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqeh1m9n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1945125Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1945131Z 
2023-01-11T22:03:10.1945206Z Running tests...
2023-01-11T22:03:10.1945405Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1945644Z   test_async_function_with_future_ctor_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8021
2023-01-11T22:03:10.1945801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8022
2023-01-11T22:03:10.1945941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8023
2023-01-11T22:03:10.1946093Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8024
2023-01-11T22:03:10.1946370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1946497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1946786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1946925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1947112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4m742jf
2023-01-11T22:03:10.1947309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4m742jf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1947571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1947696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1947980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1948120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1948344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxu5ecz1
2023-01-11T22:03:10.1948569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxu5ecz1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1948846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1948970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1949250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1949373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1949558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxq10k8s4
2023-01-11T22:03:10.1949754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxq10k8s4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1950026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1950151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1950432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1950566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1950752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdn29hkg0
2023-01-11T22:03:10.1950933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdn29hkg0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1951098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1951262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1951421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1951587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1951655Z ok (2.741s)
2023-01-11T22:03:10.1951663Z 
2023-01-11T22:03:10.1951863Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1951938Z Ran 1 test in 2.741s
2023-01-11T22:03:10.1951943Z 
2023-01-11T22:03:10.1951991Z OK
2023-01-11T22:03:10.1952007Z 
2023-01-11T22:03:10.1952079Z Generating XML reports...
2023-01-11T22:03:10.1952420Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214205.xml
2023-01-11T22:03:10.1952693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1952817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1953100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1953237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1953423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp030kp_17
2023-01-11T22:03:10.1953614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp030kp_17/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1953857Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1953875Z 
2023-01-11T22:03:10.1953936Z Running tests...
2023-01-11T22:03:10.1954132Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1954366Z   test_async_function_wrong_return_type (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8244
2023-01-11T22:03:10.1954521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8245
2023-01-11T22:03:10.1954711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8246
2023-01-11T22:03:10.1954869Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8247
2023-01-11T22:03:10.1955188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1955314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1955585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1955723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1955909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjqpyp6g
2023-01-11T22:03:10.1956107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjqpyp6g/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1956380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1956507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1956791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1956927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1957099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpev2i1ha1
2023-01-11T22:03:10.1957294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpev2i1ha1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1957568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1957695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1957975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1958113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1958299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzp827ivz
2023-01-11T22:03:10.1958494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzp827ivz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1958766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1958951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1959238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1959373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1959556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51rw5otb
2023-01-11T22:03:10.1959753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51rw5otb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1959922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1960090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1960257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1960406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1960475Z ok (2.636s)
2023-01-11T22:03:10.1960480Z 
2023-01-11T22:03:10.1960678Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1960754Z Ran 1 test in 2.636s
2023-01-11T22:03:10.1960760Z 
2023-01-11T22:03:10.1960821Z OK
2023-01-11T22:03:10.1960827Z 
2023-01-11T22:03:10.1960911Z Generating XML reports...
2023-01-11T22:03:10.1961252Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214210.xml
2023-01-11T22:03:10.1961567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1961721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1961992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1962128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1962313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5x0xb1n
2023-01-11T22:03:10.1962510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5x0xb1n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1962766Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1962772Z 
2023-01-11T22:03:10.1962844Z Running tests...
2023-01-11T22:03:10.1963044Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1963288Z   test_async_function_wrong_return_type_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8467
2023-01-11T22:03:10.1963445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8468
2023-01-11T22:03:10.1963586Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8469
2023-01-11T22:03:10.1963738Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8470
2023-01-11T22:03:10.1964012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1964136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1964418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1964553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1964743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnfkzg4i
2023-01-11T22:03:10.1964941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnfkzg4i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1965205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1965389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1965730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1965868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1966053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56nss71j
2023-01-11T22:03:10.1966247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56nss71j/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1966521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1966647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1966930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1967053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1967236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaljkc_52
2023-01-11T22:03:10.1967430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaljkc_52/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1967704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1967826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1968145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1968281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1968494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fg7eo5y
2023-01-11T22:03:10.1968675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fg7eo5y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1968841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1969004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1969168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1969330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1969398Z ok (2.647s)
2023-01-11T22:03:10.1969403Z 
2023-01-11T22:03:10.1969602Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1969676Z Ran 1 test in 2.647s
2023-01-11T22:03:10.1969682Z 
2023-01-11T22:03:10.1969743Z OK
2023-01-11T22:03:10.1969750Z 
2023-01-11T22:03:10.1969823Z Generating XML reports...
2023-01-11T22:03:10.1970166Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214215.xml
2023-01-11T22:03:10.1970440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1970569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1970852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1970989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1971174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvfl2d59x
2023-01-11T22:03:10.1971371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvfl2d59x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1971616Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1971633Z 
2023-01-11T22:03:10.1971694Z Running tests...
2023-01-11T22:03:10.1971894Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1972137Z   test_async_function_wrong_return_type_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8690
2023-01-11T22:03:10.1972292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8691
2023-01-11T22:03:10.1972446Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8692
2023-01-11T22:03:10.1972599Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8693
2023-01-11T22:03:10.1972873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1972999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1973270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1973406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1973592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2r6u0cqi
2023-01-11T22:03:10.1973787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2r6u0cqi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1974061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1974185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1974467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1974681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1974888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5e2953p
2023-01-11T22:03:10.1975084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5e2953p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1975357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1975481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1975761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1975901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1976176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptju02dks
2023-01-11T22:03:10.1976487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptju02dks/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1976796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1976908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1977192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1977327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1977509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6n01h763
2023-01-11T22:03:10.1977700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6n01h763/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1977866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1978030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1978193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1978357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1978413Z ok (2.645s)
2023-01-11T22:03:10.1978418Z 
2023-01-11T22:03:10.1978616Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1978693Z Ran 1 test in 2.645s
2023-01-11T22:03:10.1978699Z 
2023-01-11T22:03:10.1978759Z OK
2023-01-11T22:03:10.1978764Z 
2023-01-11T22:03:10.1978847Z Generating XML reports...
2023-01-11T22:03:10.1979190Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214219.xml
2023-01-11T22:03:10.1979465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1979594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1979861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1979998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1980181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_s7ybdi
2023-01-11T22:03:10.1980373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_s7ybdi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1980629Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1980635Z 
2023-01-11T22:03:10.1980707Z Running tests...
2023-01-11T22:03:10.1980905Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1981141Z   test_async_record_function_cbs_jit_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8913
2023-01-11T22:03:10.1981336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8914
2023-01-11T22:03:10.1981505Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8915
2023-01-11T22:03:10.1981808Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8916
2023-01-11T22:03:10.1982136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1982261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1982541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1982679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1982864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuy84lw2b
2023-01-11T22:03:10.1983061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuy84lw2b/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1983324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1983450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1983733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1983868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1984053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkxn9aim
2023-01-11T22:03:10.1984249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkxn9aim/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1984527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1984652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1984936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1985061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1985246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_fwxy3r
2023-01-11T22:03:10.1985519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1985642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1985926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1986060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1986246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvkmo480
2023-01-11T22:03:10.1986444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_fwxy3r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1986623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvkmo480/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1986790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1986954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1987120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1987281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1987350Z ok (2.643s)
2023-01-11T22:03:10.1987356Z 
2023-01-11T22:03:10.1987558Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1987634Z Ran 1 test in 2.643s
2023-01-11T22:03:10.1987640Z 
2023-01-11T22:03:10.1987699Z OK
2023-01-11T22:03:10.1987776Z 
2023-01-11T22:03:10.1987849Z Generating XML reports...
2023-01-11T22:03:10.1988191Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214224.xml
2023-01-11T22:03:10.1988505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1988635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1988919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1989055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1989242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ih91vua
2023-01-11T22:03:10.1989437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ih91vua/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1989679Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1989700Z 
2023-01-11T22:03:10.1989761Z Running tests...
2023-01-11T22:03:10.1989962Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1990209Z   test_async_record_function_double_end_callbacks (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9136
2023-01-11T22:03:10.1990366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9137
2023-01-11T22:03:10.1990520Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9138
2023-01-11T22:03:10.1990673Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9139
2023-01-11T22:03:10.1990952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1991077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1991352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1991490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1991678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnr4x716
2023-01-11T22:03:10.1991874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnr4x716/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1992149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1992274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1992558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1992693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1992878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinty33bq
2023-01-11T22:03:10.1993062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinty33bq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1993336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1993459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1993741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1993876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1994060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbao670m8
2023-01-11T22:03:10.1994253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbao670m8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1994524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1994670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1994977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1995113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1995297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90cuufxz
2023-01-11T22:03:10.1995492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90cuufxz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1995659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.1995823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.1995986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.1996151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.1996206Z ok (3.646s)
2023-01-11T22:03:10.1996212Z 
2023-01-11T22:03:10.1996412Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1996487Z Ran 1 test in 3.646s
2023-01-11T22:03:10.1996492Z 
2023-01-11T22:03:10.1996552Z OK
2023-01-11T22:03:10.1996558Z 
2023-01-11T22:03:10.1996642Z Generating XML reports...
2023-01-11T22:03:10.1996986Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214228.xml
2023-01-11T22:03:10.1997264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1997389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.1997659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.1997800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.1997985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10ixuapd
2023-01-11T22:03:10.1998185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10ixuapd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.1998439Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.1998444Z 
2023-01-11T22:03:10.1998517Z Running tests...
2023-01-11T22:03:10.1998715Z ----------------------------------------------------------------------
2023-01-11T22:03:10.1999038Z   test_async_record_function_legacy (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9359
2023-01-11T22:03:10.1999196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9360
2023-01-11T22:03:10.1999336Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9361
2023-01-11T22:03:10.1999492Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9362
2023-01-11T22:03:10.1999771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.1999896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2000181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2000317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2000504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb05zv3my
2023-01-11T22:03:10.2000701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb05zv3my/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2000962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2001129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2001412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2001592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2001780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawxtzvgl
2023-01-11T22:03:10.2001977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawxtzvgl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2002250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2002374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2002655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2002780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2002966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeprc464v
2023-01-11T22:03:10.2003164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeprc464v/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2003436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2003559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2003838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2003973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2004157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4zpb5flu
2023-01-11T22:03:10.2004341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4zpb5flu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2004508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2004673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2004839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2005001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2005069Z ok (3.648s)
2023-01-11T22:03:10.2005074Z 
2023-01-11T22:03:10.2005272Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2005347Z Ran 1 test in 3.648s
2023-01-11T22:03:10.2005353Z 
2023-01-11T22:03:10.2005416Z OK
2023-01-11T22:03:10.2005421Z 
2023-01-11T22:03:10.2005492Z Generating XML reports...
2023-01-11T22:03:10.2005838Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214234.xml
2023-01-11T22:03:10.2006117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2006241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2006523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2006658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2006842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4h_b6nxs
2023-01-11T22:03:10.2007037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4h_b6nxs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2007276Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2007294Z 
2023-01-11T22:03:10.2007354Z Running tests...
2023-01-11T22:03:10.2007550Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2007814Z   test_async_static_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9582
2023-01-11T22:03:10.2007999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9583
2023-01-11T22:03:10.2008156Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9584
2023-01-11T22:03:10.2008308Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9585
2023-01-11T22:03:10.2008584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2008709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2008977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2009111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2009300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjz00vpgh
2023-01-11T22:03:10.2009499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjz00vpgh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2009774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2009898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2010179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2010314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2010499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29o9zg44
2023-01-11T22:03:10.2010679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29o9zg44/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2010948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2011074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2011356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2011493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2011679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpho3tp5pq
2023-01-11T22:03:10.2011875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpho3tp5pq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2012148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2012258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2012540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2012677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2012864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9cbe4h3
2023-01-11T22:03:10.2013057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9cbe4h3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2013222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2013389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2013553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2013715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2013769Z ok (2.642s)
2023-01-11T22:03:10.2013775Z 
2023-01-11T22:03:10.2013972Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2014083Z Ran 1 test in 2.642s
2023-01-11T22:03:10.2014089Z 
2023-01-11T22:03:10.2014151Z OK
2023-01-11T22:03:10.2014157Z 
2023-01-11T22:03:10.2014241Z Generating XML reports...
2023-01-11T22:03:10.2014608Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214239.xml
2023-01-11T22:03:10.2014885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2015010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2015278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2015415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2015600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvuhe6g_d
2023-01-11T22:03:10.2015797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvuhe6g_d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2016054Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2016062Z 
2023-01-11T22:03:10.2016137Z Running tests...
2023-01-11T22:03:10.2016333Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2016566Z   test_async_static_method_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9805
2023-01-11T22:03:10.2016722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9806
2023-01-11T22:03:10.2016862Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9807
2023-01-11T22:03:10.2017014Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9808
2023-01-11T22:03:10.2017293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2017419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2017703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2017840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2018030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc291bo3f
2023-01-11T22:03:10.2018226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc291bo3f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2018485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2018609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2018890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2019027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2019213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwonmstz
2023-01-11T22:03:10.2019412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwonmstz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2019687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2019810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2020089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2020211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2020395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjetp34jp
2023-01-11T22:03:10.2020590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjetp34jp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2020897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2021050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2021335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2021469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2021795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppzg81h_0
2023-01-11T22:03:10.2021980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppzg81h_0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2022146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2022311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2022476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2022640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2022711Z ok (2.645s)
2023-01-11T22:03:10.2022717Z 
2023-01-11T22:03:10.2022917Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2022994Z Ran 1 test in 2.645s
2023-01-11T22:03:10.2022999Z 
2023-01-11T22:03:10.2023066Z OK
2023-01-11T22:03:10.2023071Z 
2023-01-11T22:03:10.2023143Z Generating XML reports...
2023-01-11T22:03:10.2023488Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214244.xml
2023-01-11T22:03:10.2023766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2023891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2024175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2024316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2024504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6nw5js1
2023-01-11T22:03:10.2024700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6nw5js1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2024943Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2024961Z 
2023-01-11T22:03:10.2025022Z Running tests...
2023-01-11T22:03:10.2025220Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2025450Z   test_build_rpc_profiling_key (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10028
2023-01-11T22:03:10.2025609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10029
2023-01-11T22:03:10.2025769Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10030
2023-01-11T22:03:10.2025925Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10031
2023-01-11T22:03:10.2026201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2026326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2026596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2026732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2026917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjbdt2wr
2023-01-11T22:03:10.2027112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjbdt2wr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2027387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2027565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2027887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2028027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2028211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9kn4_vzx
2023-01-11T22:03:10.2028392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9kn4_vzx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2028665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2028788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2029068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2029207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2029396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmrhbip6
2023-01-11T22:03:10.2029592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmrhbip6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2029866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2029977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2030257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2030393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2030576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69l7jfqo
2023-01-11T22:03:10.2030775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69l7jfqo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2030942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2031107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2031270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2031431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2031486Z ok (2.347s)
2023-01-11T22:03:10.2031492Z 
2023-01-11T22:03:10.2031688Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2031763Z Ran 1 test in 2.347s
2023-01-11T22:03:10.2031769Z 
2023-01-11T22:03:10.2031830Z OK
2023-01-11T22:03:10.2031836Z 
2023-01-11T22:03:10.2031919Z Generating XML reports...
2023-01-11T22:03:10.2032257Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214249.xml
2023-01-11T22:03:10.2032536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2032659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2032924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2033060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2033243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuh_lurcu
2023-01-11T22:03:10.2033439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuh_lurcu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2033694Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2033700Z 
2023-01-11T22:03:10.2033810Z Running tests...
2023-01-11T22:03:10.2034011Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2034261Z   test_builtin_remote_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10083
2023-01-11T22:03:10.2034423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10084
2023-01-11T22:03:10.2034566Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10085
2023-01-11T22:03:10.2034721Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10086
2023-01-11T22:03:10.2034994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2035118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2035401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2035540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2035724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8atyd9m3
2023-01-11T22:03:10.2035920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8atyd9m3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2036177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2036301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2036578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2036715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2036899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujyfghnp
2023-01-11T22:03:10.2037171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2037297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2037579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2037713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2037885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzee2hi1y
2023-01-11T22:03:10.2038160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2038282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2038563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2038697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2038951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp029uhc_b
2023-01-11T22:03:10.2039156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujyfghnp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2039349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzee2hi1y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2039525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp029uhc_b/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2039692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2039858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2040019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2040183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2040253Z ok (2.644s)
2023-01-11T22:03:10.2040302Z 
2023-01-11T22:03:10.2040511Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2040587Z Ran 1 test in 2.645s
2023-01-11T22:03:10.2040592Z 
2023-01-11T22:03:10.2040682Z OK
2023-01-11T22:03:10.2040687Z 
2023-01-11T22:03:10.2040760Z Generating XML reports...
2023-01-11T22:03:10.2041097Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214253.xml
2023-01-11T22:03:10.2041371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2041494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2041774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2041910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2042096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_od0upx2
2023-01-11T22:03:10.2042289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_od0upx2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2042534Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2042551Z 
2023-01-11T22:03:10.2042612Z Running tests...
2023-01-11T22:03:10.2042807Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2043035Z   test_builtin_remote_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10306
2023-01-11T22:03:10.2043192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10307
2023-01-11T22:03:10.2043348Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10308
2023-01-11T22:03:10.2043505Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10309
2023-01-11T22:03:10.2043779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2043903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2044174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2044311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2044498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwu7lmxym
2023-01-11T22:03:10.2044695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwu7lmxym/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2044970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2045094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2045378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2045516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2045703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufhysnob
2023-01-11T22:03:10.2045885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufhysnob/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2046157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2046281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2046569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2046704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2046889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozva0t27
2023-01-11T22:03:10.2047119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozva0t27/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2047435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2047547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2047830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2047965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2048151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1plvj_g3
2023-01-11T22:03:10.2048346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1plvj_g3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2048510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2048679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2048845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2049009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2049064Z ok (2.641s)
2023-01-11T22:03:10.2049069Z 
2023-01-11T22:03:10.2049267Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2049343Z Ran 1 test in 2.641s
2023-01-11T22:03:10.2049348Z 
2023-01-11T22:03:10.2049409Z OK
2023-01-11T22:03:10.2049414Z 
2023-01-11T22:03:10.2049497Z Generating XML reports...
2023-01-11T22:03:10.2049841Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214257.xml
2023-01-11T22:03:10.2050116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2050245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2050517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2050654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2050841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpra2p4k16
2023-01-11T22:03:10.2051038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpra2p4k16/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2051300Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2051305Z 
2023-01-11T22:03:10.2051379Z Running tests...
2023-01-11T22:03:10.2051576Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2051699Z   test_call_method_on_rref (__main__.TensorPipeRpcTest)
2023-01-11T22:03:10.2051932Z Tests that it is possible to call an instance method on a remote objet ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10529
2023-01-11T22:03:10.2052081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10530
2023-01-11T22:03:10.2052236Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10531
2023-01-11T22:03:10.2052389Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10532
2023-01-11T22:03:10.2052663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2052788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2053071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2053205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2053389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5zhjezn8
2023-01-11T22:03:10.2053607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5zhjezn8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2053910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2054035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2054317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2054451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2054639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdp5enfkh
2023-01-11T22:03:10.2054836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdp5enfkh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2055106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2055231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2055500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2055637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2055823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzx3vgskz
2023-01-11T22:03:10.2056017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzx3vgskz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2056286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2056409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2056689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2056826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2056997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsuzk54nn
2023-01-11T22:03:10.2057194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsuzk54nn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2057359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2057527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2057689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2057852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2057919Z ok (2.754s)
2023-01-11T22:03:10.2057925Z 
2023-01-11T22:03:10.2058122Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2058199Z Ran 1 test in 2.754s
2023-01-11T22:03:10.2058205Z 
2023-01-11T22:03:10.2058253Z OK
2023-01-11T22:03:10.2058258Z 
2023-01-11T22:03:10.2058347Z Generating XML reports...
2023-01-11T22:03:10.2058689Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214302.xml
2023-01-11T22:03:10.2058961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2059086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2059368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2059504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2059686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2o_ctgb1
2023-01-11T22:03:10.2059866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2o_ctgb1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2060154Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2060160Z 
2023-01-11T22:03:10.2060263Z Running tests...
2023-01-11T22:03:10.2060462Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2060687Z   test_callback_chain (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10752
2023-01-11T22:03:10.2060845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10753
2023-01-11T22:03:10.2061000Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10754
2023-01-11T22:03:10.2061153Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10755
2023-01-11T22:03:10.2061429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2061541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2062046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2062223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2062412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7mjhs0e
2023-01-11T22:03:10.2062693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2062819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2063102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2063238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2063411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuc5cbjsk
2023-01-11T22:03:10.2063616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7mjhs0e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2063810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuc5cbjsk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2064085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2064209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2064492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2064627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2064811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvnk3mmt
2023-01-11T22:03:10.2065006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvnk3mmt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2065267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2065393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2065675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2065810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2065996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6r40037o
2023-01-11T22:03:10.2066194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6r40037o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2066357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2066523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2066686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2066906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2066974Z ok (2.550s)
2023-01-11T22:03:10.2067016Z 
2023-01-11T22:03:10.2067220Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2067297Z Ran 1 test in 2.550s
2023-01-11T22:03:10.2067303Z 
2023-01-11T22:03:10.2067363Z OK
2023-01-11T22:03:10.2067369Z 
2023-01-11T22:03:10.2067453Z Generating XML reports...
2023-01-11T22:03:10.2067798Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214307.xml
2023-01-11T22:03:10.2068075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2068186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2068471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2068609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2068801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnk4e4ogp
2023-01-11T22:03:10.2068997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnk4e4ogp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2069253Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2069259Z 
2023-01-11T22:03:10.2069331Z Running tests...
2023-01-11T22:03:10.2069529Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2069753Z   test_callback_in_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10975
2023-01-11T22:03:10.2069900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10976
2023-01-11T22:03:10.2070059Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10977
2023-01-11T22:03:10.2070215Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10978
2023-01-11T22:03:10.2070494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2070620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2070905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2071039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2071228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpct86ho_p
2023-01-11T22:03:10.2071412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpct86ho_p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2071687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2071813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2072098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2072232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2072417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawwynokd
2023-01-11T22:03:10.2072613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawwynokd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2072885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2073009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2073277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2073447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2073701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplk60q78t
2023-01-11T22:03:10.2073897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplk60q78t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2074172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2074296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2074576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2074712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2074883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4q6zlqi
2023-01-11T22:03:10.2075076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4q6zlqi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2075243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2075410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2075574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2075738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2075806Z ok (2.646s)
2023-01-11T22:03:10.2075814Z 
2023-01-11T22:03:10.2076011Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2076087Z Ran 1 test in 2.646s
2023-01-11T22:03:10.2076093Z 
2023-01-11T22:03:10.2076141Z OK
2023-01-11T22:03:10.2076146Z 
2023-01-11T22:03:10.2076231Z Generating XML reports...
2023-01-11T22:03:10.2076571Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214311.xml
2023-01-11T22:03:10.2076850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2076978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2077257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2077395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2077576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_i2aok93
2023-01-11T22:03:10.2077755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_i2aok93/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2078012Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2078018Z 
2023-01-11T22:03:10.2078091Z Running tests...
2023-01-11T22:03:10.2078290Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2078512Z   test_callback_multi (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11198
2023-01-11T22:03:10.2078675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11199
2023-01-11T22:03:10.2078893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11200
2023-01-11T22:03:10.2079050Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11201
2023-01-11T22:03:10.2079329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2079440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2079728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2079866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2080090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe13hl8uz
2023-01-11T22:03:10.2080393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2080521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2080805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2080944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2081118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp988bjh94
2023-01-11T22:03:10.2081316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe13hl8uz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2081504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp988bjh94/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2081780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2081907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2082186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2082321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2082505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9rhm_ab
2023-01-11T22:03:10.2082698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9rhm_ab/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2082960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2083085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2083365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2083503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2083691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxchowsj5
2023-01-11T22:03:10.2083886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxchowsj5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2084052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2084217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2084367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2084529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2084601Z ok (2.649s)
2023-01-11T22:03:10.2084607Z 
2023-01-11T22:03:10.2084806Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2084882Z Ran 1 test in 2.650s
2023-01-11T22:03:10.2084888Z 
2023-01-11T22:03:10.2084947Z OK
2023-01-11T22:03:10.2084952Z 
2023-01-11T22:03:10.2085039Z Generating XML reports...
2023-01-11T22:03:10.2085380Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214316.xml
2023-01-11T22:03:10.2085657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2085768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2086052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2086189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2086375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgulgplma
2023-01-11T22:03:10.2086611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgulgplma/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2086904Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2086910Z 
2023-01-11T22:03:10.2086986Z Running tests...
2023-01-11T22:03:10.2087187Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2087407Z   test_callback_none (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11421
2023-01-11T22:03:10.2087552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11422
2023-01-11T22:03:10.2087708Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11423
2023-01-11T22:03:10.2087862Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11424
2023-01-11T22:03:10.2088138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2088266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2088552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2088686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2088871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyjz8046
2023-01-11T22:03:10.2089053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyjz8046/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2089325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2089448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2089730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2089868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2090056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2c2fhe_
2023-01-11T22:03:10.2090254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2c2fhe_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2090527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2090651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2090918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2091052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2091238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkju2fp1
2023-01-11T22:03:10.2091433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkju2fp1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2091707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2091831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2092113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2092249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2092420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqft9afel
2023-01-11T22:03:10.2092615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqft9afel/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2092779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2092943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2093147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2093358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2093452Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.2093575Z ValueError('Expected error')
2023-01-11T22:03:10.2093654Z Traceback (most recent call last):
2023-01-11T22:03:10.2093919Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2094056Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2094342Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.2094435Z     raise ValueError(expected_err)
2023-01-11T22:03:10.2094523Z ValueError: Expected error
2023-01-11T22:03:10.2094529Z 
2023-01-11T22:03:10.2094622Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.2094742Z ValueError('Expected error')
2023-01-11T22:03:10.2094820Z Traceback (most recent call last):
2023-01-11T22:03:10.2095081Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2095217Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2095496Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.2095589Z     raise ValueError(expected_err)
2023-01-11T22:03:10.2095674Z ValueError: Expected error
2023-01-11T22:03:10.2095680Z 
2023-01-11T22:03:10.2095771Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.2095889Z ValueError('Expected error')
2023-01-11T22:03:10.2095967Z Traceback (most recent call last):
2023-01-11T22:03:10.2096219Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2096354Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2096634Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.2096726Z     raise ValueError(expected_err)
2023-01-11T22:03:10.2096811Z ValueError: Expected error
2023-01-11T22:03:10.2096816Z 
2023-01-11T22:03:10.2096906Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.2097024Z ValueError('Expected error')
2023-01-11T22:03:10.2097101Z Traceback (most recent call last):
2023-01-11T22:03:10.2097354Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2097489Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2097767Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.2097861Z     raise ValueError(expected_err)
2023-01-11T22:03:10.2097946Z ValueError: Expected error
2023-01-11T22:03:10.2097951Z 
2023-01-11T22:03:10.2098017Z ok (2.644s)
2023-01-11T22:03:10.2098023Z 
2023-01-11T22:03:10.2098224Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2098287Z Ran 1 test in 2.644s
2023-01-11T22:03:10.2098293Z 
2023-01-11T22:03:10.2098354Z OK
2023-01-11T22:03:10.2098359Z 
2023-01-11T22:03:10.2098442Z Generating XML reports...
2023-01-11T22:03:10.2098783Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214320.xml
2023-01-11T22:03:10.2099057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2099183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2099468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2099640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2099849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpse7ucsdl
2023-01-11T22:03:10.2100050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpse7ucsdl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2100306Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2100312Z 
2023-01-11T22:03:10.2100385Z Running tests...
2023-01-11T22:03:10.2100583Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2100804Z   test_callback_simple (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11644
2023-01-11T22:03:10.2100961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11645
2023-01-11T22:03:10.2101116Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11646
2023-01-11T22:03:10.2101273Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11647
2023-01-11T22:03:10.2101537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2101802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2102087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2102225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2102413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp018r7d7c
2023-01-11T22:03:10.2102607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp018r7d7c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2102882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2103010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2103282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2103417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2103602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7244pnm5
2023-01-11T22:03:10.2103798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7244pnm5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2104076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2104200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2104487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2104623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2104811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9it4o_0
2023-01-11T22:03:10.2104997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9it4o_0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2105272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2105395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2105677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2105813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2105999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1z4tvfp
2023-01-11T22:03:10.2106196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1z4tvfp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2106422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2106611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2106778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2106940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2107008Z ok (2.749s)
2023-01-11T22:03:10.2107014Z 
2023-01-11T22:03:10.2107215Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2107291Z Ran 1 test in 2.749s
2023-01-11T22:03:10.2107296Z 
2023-01-11T22:03:10.2107359Z OK
2023-01-11T22:03:10.2107364Z 
2023-01-11T22:03:10.2107448Z Generating XML reports...
2023-01-11T22:03:10.2107791Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214325.xml
2023-01-11T22:03:10.2108056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2108183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2108465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2108602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2108788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjmzp9xm
2023-01-11T22:03:10.2108985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjmzp9xm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2109242Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2109248Z 
2023-01-11T22:03:10.2109320Z Running tests...
2023-01-11T22:03:10.2109505Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2109731Z   test_callback_with_error (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11867
2023-01-11T22:03:10.2109890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11868
2023-01-11T22:03:10.2110048Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11869
2023-01-11T22:03:10.2110202Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11870
2023-01-11T22:03:10.2110479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2110603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2110885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2111022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2111196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8bxzv59d
2023-01-11T22:03:10.2111391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8bxzv59d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2111665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2111789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2112069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2112206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2112392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmyxg9wp4
2023-01-11T22:03:10.2112588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmyxg9wp4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2112858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2113004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2113317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2113454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2113637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2om5qjiw
2023-01-11T22:03:10.2113833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2om5qjiw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2114106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2114229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2114509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2114633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2114819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvedgwpu
2023-01-11T22:03:10.2115013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvedgwpu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2115178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2115344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2115509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2115672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2115764Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.2115873Z ValueError('Expected error')
2023-01-11T22:03:10.2115967Z Traceback (most recent call last):
2023-01-11T22:03:10.2116230Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2116365Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2116652Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.2116743Z     raise ValueError(expected_err)
2023-01-11T22:03:10.2116830Z ValueError: Expected error
2023-01-11T22:03:10.2116835Z 
2023-01-11T22:03:10.2116925Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.2117070Z ValueError('Expected error')
2023-01-11T22:03:10.2117197Z Traceback (most recent call last):
2023-01-11T22:03:10.2117511Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2117643Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2117926Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.2118019Z     raise ValueError(expected_err)
2023-01-11T22:03:10.2118108Z ValueError: Expected error
2023-01-11T22:03:10.2118114Z 
2023-01-11T22:03:10.2118207Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.2118313Z ValueError('Expected error')
2023-01-11T22:03:10.2118404Z Traceback (most recent call last):
2023-01-11T22:03:10.2118659Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2118791Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2119132Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.2119226Z     raise ValueError(expected_err)
2023-01-11T22:03:10.2119311Z ValueError: Expected error
2023-01-11T22:03:10.2119316Z 
2023-01-11T22:03:10.2119454Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.2119561Z ValueError('Expected error')
2023-01-11T22:03:10.2119652Z Traceback (most recent call last):
2023-01-11T22:03:10.2119934Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2120068Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2120347Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.2120438Z     raise ValueError(expected_err)
2023-01-11T22:03:10.2120523Z ValueError: Expected error
2023-01-11T22:03:10.2120528Z 
2023-01-11T22:03:10.2120582Z ok (2.644s)
2023-01-11T22:03:10.2120600Z 
2023-01-11T22:03:10.2120783Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2120857Z Ran 1 test in 2.644s
2023-01-11T22:03:10.2120863Z 
2023-01-11T22:03:10.2120923Z OK
2023-01-11T22:03:10.2120930Z 
2023-01-11T22:03:10.2121014Z Generating XML reports...
2023-01-11T22:03:10.2121354Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214330.xml
2023-01-11T22:03:10.2121630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2121756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2122036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2122160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2122348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhozrxqu
2023-01-11T22:03:10.2122545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhozrxqu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2122801Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2122810Z 
2023-01-11T22:03:10.2122883Z Running tests...
2023-01-11T22:03:10.2123083Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2123307Z   test_callback_with_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12090
2023-01-11T22:03:10.2123465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12091
2023-01-11T22:03:10.2123620Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12092
2023-01-11T22:03:10.2123763Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12093
2023-01-11T22:03:10.2124034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2124156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2124438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2124575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2124763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg92azf6t
2023-01-11T22:03:10.2124957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg92azf6t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2125230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2125341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2125623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2125758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2125941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgo61_s7h
2023-01-11T22:03:10.2126174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgo61_s7h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2126475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2126601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2126884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2127020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2127192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgp9kqxjo
2023-01-11T22:03:10.2127387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgp9kqxjo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2127661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2127829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2128250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2128387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2128573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmz5dt5fs
2023-01-11T22:03:10.2128770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmz5dt5fs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2128922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2129087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2129252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2129415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2129485Z ok (2.642s)
2023-01-11T22:03:10.2129491Z 
2023-01-11T22:03:10.2129695Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2129771Z Ran 1 test in 2.643s
2023-01-11T22:03:10.2129777Z 
2023-01-11T22:03:10.2129836Z OK
2023-01-11T22:03:10.2129841Z 
2023-01-11T22:03:10.2129925Z Generating XML reports...
2023-01-11T22:03:10.2130253Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214334.xml
2023-01-11T22:03:10.2130529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2130654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2130936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2131072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2131261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ltkbm35
2023-01-11T22:03:10.2131459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ltkbm35/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2131713Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2131718Z 
2023-01-11T22:03:10.2131792Z Running tests...
2023-01-11T22:03:10.2131975Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2132200Z   test_callback_wrong_arg_num (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12313
2023-01-11T22:03:10.2132358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12314
2023-01-11T22:03:10.2132513Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12315
2023-01-11T22:03:10.2132712Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12316
2023-01-11T22:03:10.2133020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2133147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2133429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2133552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2133737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoa5tb16c
2023-01-11T22:03:10.2134011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2134133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2134416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2134553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2134739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0r7ji_0l
2023-01-11T22:03:10.2134934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoa5tb16c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2135110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0r7ji_0l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2135380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2135503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2135782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2135916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2136102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpw8h80j
2023-01-11T22:03:10.2136297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpw8h80j/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2136569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2136693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2136961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2137098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2137282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpus5gi2uv
2023-01-11T22:03:10.2137476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpus5gi2uv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2137642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2137807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2137971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2138134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2138189Z ok (2.742s)
2023-01-11T22:03:10.2138206Z 
2023-01-11T22:03:10.2138390Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2138465Z Ran 1 test in 2.742s
2023-01-11T22:03:10.2138470Z 
2023-01-11T22:03:10.2138530Z OK
2023-01-11T22:03:10.2138535Z 
2023-01-11T22:03:10.2138619Z Generating XML reports...
2023-01-11T22:03:10.2138960Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214339.xml
2023-01-11T22:03:10.2139234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2139395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2139720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2139845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2140031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq83fyu_1
2023-01-11T22:03:10.2140227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq83fyu_1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2140486Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2140491Z 
2023-01-11T22:03:10.2140564Z Running tests...
2023-01-11T22:03:10.2140760Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2140992Z   test_callback_wrong_arg_type (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12536
2023-01-11T22:03:10.2141151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12537
2023-01-11T22:03:10.2141308Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12538
2023-01-11T22:03:10.2141449Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12539
2023-01-11T22:03:10.2141847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2141974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2142260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2142398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2142586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkoc5pa_s
2023-01-11T22:03:10.2142791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkoc5pa_s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2143069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2143180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2143463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2143598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2143784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19t9vxaz
2023-01-11T22:03:10.2143979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19t9vxaz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2144250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2144379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2144662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2144796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2144968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoih7zsp1
2023-01-11T22:03:10.2145167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoih7zsp1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2145436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2145561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2145840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2146031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2146248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcntrc6x2
2023-01-11T22:03:10.2146445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcntrc6x2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2146598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2146764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2146926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2147090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2147158Z ok (2.642s)
2023-01-11T22:03:10.2147165Z 
2023-01-11T22:03:10.2147369Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2147448Z Ran 1 test in 2.642s
2023-01-11T22:03:10.2147453Z 
2023-01-11T22:03:10.2147514Z OK
2023-01-11T22:03:10.2147519Z 
2023-01-11T22:03:10.2147602Z Generating XML reports...
2023-01-11T22:03:10.2147931Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214344.xml
2023-01-11T22:03:10.2148209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2148332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2148615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2148750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2148936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqfe8nhp5
2023-01-11T22:03:10.2149131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqfe8nhp5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2149388Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2149394Z 
2023-01-11T22:03:10.2149470Z Running tests...
2023-01-11T22:03:10.2149653Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2149889Z   test_cannot_infer_backend_from_options (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12759
2023-01-11T22:03:10.2150049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12760
2023-01-11T22:03:10.2150206Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12761
2023-01-11T22:03:10.2150359Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12762
2023-01-11T22:03:10.2150633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2150758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2151040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2151163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2151347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpidw7itju
2023-01-11T22:03:10.2151542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpidw7itju/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2151815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2151939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2152218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2152356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2152571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kujxn22
2023-01-11T22:03:10.2152793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kujxn22/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2153052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2153176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2153455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2153589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2153773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7hd_omvh
2023-01-11T22:03:10.2153963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7hd_omvh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2154235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2154360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2154625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2154760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2154944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbrjsv3yo
2023-01-11T22:03:10.2155139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbrjsv3yo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2155302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2155467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2155632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2155794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2155850Z ok (2.341s)
2023-01-11T22:03:10.2155867Z 
2023-01-11T22:03:10.2156051Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2156127Z Ran 1 test in 2.342s
2023-01-11T22:03:10.2156132Z 
2023-01-11T22:03:10.2156192Z OK
2023-01-11T22:03:10.2156197Z 
2023-01-11T22:03:10.2156281Z Generating XML reports...
2023-01-11T22:03:10.2156621Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214348.xml
2023-01-11T22:03:10.2156893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2157017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2157297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2157422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2157607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_szj1kc
2023-01-11T22:03:10.2157799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_szj1kc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2158055Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2158061Z 
2023-01-11T22:03:10.2158133Z Running tests...
2023-01-11T22:03:10.2158332Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2158489Z   test_custom_exception_throw_during_reconstruction (__main__.TensorPipeRpcTest)
2023-01-11T22:03:10.2158723Z Test that we still throw info about the remote side exception even when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12814
2023-01-11T22:03:10.2158987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12815
2023-01-11T22:03:10.2159166Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12816
2023-01-11T22:03:10.2159324Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12817
2023-01-11T22:03:10.2159602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2159726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2160011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2160147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2160333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9kfzhe5h
2023-01-11T22:03:10.2160528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9kfzhe5h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2160792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2160919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2161197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2161333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2161520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj80i1rp_
2023-01-11T22:03:10.2161717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj80i1rp_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2161990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2162113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2162395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2162518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2162702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbropvs2
2023-01-11T22:03:10.2162892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbropvs2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2163167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2163290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2163567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2163700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2163885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppy0k283t
2023-01-11T22:03:10.2164082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppy0k283t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2164248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2164416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2164567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2164728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2164910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.2165088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.2165263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.2165480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.2165817Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2166123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2166417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2166699Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2167118Z Got msg RuntimeError: Failed to create original exception type. Error msg was __init__() missing 1 required positional argument: 'msg' Original exception on remote side was On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.2167237Z CustomException('foo')
2023-01-11T22:03:10.2167333Z Traceback (most recent call last):
2023-01-11T22:03:10.2167600Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2167740Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2168041Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 409, in custom_raise_func
2023-01-11T22:03:10.2168145Z     raise CustomException(True, "foo")
2023-01-11T22:03:10.2168307Z torch.testing._internal.distributed.rpc.rpc_test.CustomException: foo
2023-01-11T22:03:10.2168313Z 
2023-01-11T22:03:10.2168318Z 
2023-01-11T22:03:10.2168367Z At:
2023-01-11T22:03:10.2168617Z   /opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(231): _handle_exception
2023-01-11T22:03:10.2168623Z 
2023-01-11T22:03:10.2169039Z Got msg RuntimeError: Failed to create original exception type. Error msg was __init__() missing 1 required positional argument: 'msg' Original exception on remote side was On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.2169160Z CustomException('foo')
2023-01-11T22:03:10.2169256Z Traceback (most recent call last):
2023-01-11T22:03:10.2169517Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2169655Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2169952Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 409, in custom_raise_func
2023-01-11T22:03:10.2170054Z     raise CustomException(True, "foo")
2023-01-11T22:03:10.2170203Z torch.testing._internal.distributed.rpc.rpc_test.CustomException: foo
2023-01-11T22:03:10.2170209Z 
2023-01-11T22:03:10.2170230Z 
2023-01-11T22:03:10.2170277Z At:
2023-01-11T22:03:10.2170526Z   /opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(231): _handle_exception
2023-01-11T22:03:10.2170533Z 
2023-01-11T22:03:10.2170951Z Got msg RuntimeError: Failed to create original exception type. Error msg was __init__() missing 1 required positional argument: 'msg' Original exception on remote side was On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.2171067Z CustomException('foo')
2023-01-11T22:03:10.2171164Z Traceback (most recent call last):
2023-01-11T22:03:10.2171428Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2171563Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2171864Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 409, in custom_raise_func
2023-01-11T22:03:10.2171951Z     raise CustomException(True, "foo")
2023-01-11T22:03:10.2172114Z torch.testing._internal.distributed.rpc.rpc_test.CustomException: foo
2023-01-11T22:03:10.2172151Z 
2023-01-11T22:03:10.2172156Z 
2023-01-11T22:03:10.2172219Z At:
2023-01-11T22:03:10.2172466Z   /opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(231): _handle_exception
2023-01-11T22:03:10.2172499Z 
2023-01-11T22:03:10.2172593Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.2172707Z CustomException('foo')
2023-01-11T22:03:10.2172801Z Traceback (most recent call last):
2023-01-11T22:03:10.2173060Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2173180Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2173471Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 409, in custom_raise_func
2023-01-11T22:03:10.2173570Z     raise CustomException(True, "foo")
2023-01-11T22:03:10.2173731Z torch.testing._internal.distributed.rpc.rpc_test.CustomException: foo
2023-01-11T22:03:10.2173736Z 
2023-01-11T22:03:10.2173830Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.2173943Z CustomException('foo')
2023-01-11T22:03:10.2174041Z Traceback (most recent call last):
2023-01-11T22:03:10.2174299Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2174417Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2174704Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 409, in custom_raise_func
2023-01-11T22:03:10.2174803Z     raise CustomException(True, "foo")
2023-01-11T22:03:10.2174961Z torch.testing._internal.distributed.rpc.rpc_test.CustomException: foo
2023-01-11T22:03:10.2174966Z 
2023-01-11T22:03:10.2175056Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.2175169Z CustomException('foo')
2023-01-11T22:03:10.2175262Z Traceback (most recent call last):
2023-01-11T22:03:10.2175514Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2175634Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2175924Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 409, in custom_raise_func
2023-01-11T22:03:10.2176022Z     raise CustomException(True, "foo")
2023-01-11T22:03:10.2176179Z torch.testing._internal.distributed.rpc.rpc_test.CustomException: foo
2023-01-11T22:03:10.2176185Z 
2023-01-11T22:03:10.2176250Z ok (2.739s)
2023-01-11T22:03:10.2176255Z 
2023-01-11T22:03:10.2176451Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2176525Z Ran 1 test in 2.739s
2023-01-11T22:03:10.2176530Z 
2023-01-11T22:03:10.2176589Z OK
2023-01-11T22:03:10.2176594Z 
2023-01-11T22:03:10.2176677Z Generating XML reports...
2023-01-11T22:03:10.2177004Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214352.xml
2023-01-11T22:03:10.2177278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2177406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2177689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2177824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2178012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpleraewvc
2023-01-11T22:03:10.2178210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpleraewvc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2178464Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2178470Z 
2023-01-11T22:03:10.2178542Z Running tests...
2023-01-11T22:03:10.2178726Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2179012Z   test_deadlock (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13049
2023-01-11T22:03:10.2179201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13050
2023-01-11T22:03:10.2179362Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13051
2023-01-11T22:03:10.2179521Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13052
2023-01-11T22:03:10.2179800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2179926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2180210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2180334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2180526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94p7qby_
2023-01-11T22:03:10.2180726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94p7qby_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2181000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2181124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2181407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2181544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2181853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mxwfo46
2023-01-11T22:03:10.2182036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mxwfo46/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2182312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2182441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2182722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2182857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2183043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelfb6gy9
2023-01-11T22:03:10.2183240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelfb6gy9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2183514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2183638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2183906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2184044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2184230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp764qnc4x
2023-01-11T22:03:10.2184424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp764qnc4x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2184590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2184756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2184919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2185081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2185258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.2185478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.2185695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.2185872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.2186172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2186470Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2186765Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2187058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2187130Z ok (3.845s)
2023-01-11T22:03:10.2187142Z 
2023-01-11T22:03:10.2187342Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2187405Z Ran 1 test in 3.845s
2023-01-11T22:03:10.2187410Z 
2023-01-11T22:03:10.2187475Z OK
2023-01-11T22:03:10.2187481Z 
2023-01-11T22:03:10.2187570Z Generating XML reports...
2023-01-11T22:03:10.2187913Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214357.xml
2023-01-11T22:03:10.2188186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2188310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2188592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2188728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2188899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_ej3wng
2023-01-11T22:03:10.2189096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_ej3wng/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2189349Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2189355Z 
2023-01-11T22:03:10.2189429Z Running tests...
2023-01-11T22:03:10.2189628Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2189842Z   test_debug_info (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13284
2023-01-11T22:03:10.2189999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13285
2023-01-11T22:03:10.2190154Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13286
2023-01-11T22:03:10.2190307Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13287
2023-01-11T22:03:10.2190574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2190698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2190984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2191118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2191304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcol4tt8t
2023-01-11T22:03:10.2191499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcol4tt8t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2191771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2191894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2192162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2192328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2192543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcncvtj4i
2023-01-11T22:03:10.2192741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcncvtj4i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2193012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2193135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2193412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2193546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2193729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp0vo1ij1
2023-01-11T22:03:10.2193912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp0vo1ij1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2194188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2194311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2194589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2194725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2194906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2e7ujx13
2023-01-11T22:03:10.2195100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2e7ujx13/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2195263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2195417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2195579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2195742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2195809Z ok (2.741s)
2023-01-11T22:03:10.2195814Z 
2023-01-11T22:03:10.2196011Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2196086Z Ran 1 test in 2.742s
2023-01-11T22:03:10.2196092Z 
2023-01-11T22:03:10.2196153Z OK
2023-01-11T22:03:10.2196159Z 
2023-01-11T22:03:10.2196242Z Generating XML reports...
2023-01-11T22:03:10.2196582Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214403.xml
2023-01-11T22:03:10.2196841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2196966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2197250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2197386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2197572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpao7n_iws
2023-01-11T22:03:10.2197766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpao7n_iws/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2198021Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2198026Z 
2023-01-11T22:03:10.2198099Z Running tests...
2023-01-11T22:03:10.2198284Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2198414Z   test_default_timeout_used (__main__.TensorPipeRpcTest)
2023-01-11T22:03:10.2198651Z Tests that if no timeout is passed into rpc_async and rpc_sync, then the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13511
2023-01-11T22:03:10.2198953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13512
2023-01-11T22:03:10.2199124Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13513
2023-01-11T22:03:10.2199279Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13514
2023-01-11T22:03:10.2199561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2199689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2199973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2200097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2200286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tba889p
2023-01-11T22:03:10.2200484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tba889p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2200761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2200890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2201172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2201309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2201497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3hwyius
2023-01-11T22:03:10.2201679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3hwyius/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2201952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2202079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2202362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2202497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2202680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2euleu0
2023-01-11T22:03:10.2202876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2euleu0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2203148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2203274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2203544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2203680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2203865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwjux7m7
2023-01-11T22:03:10.2204064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwjux7m7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2204228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2204391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2204554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2204715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2204770Z ok (4.654s)
2023-01-11T22:03:10.2204788Z 
2023-01-11T22:03:10.2204975Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2205051Z Ran 1 test in 4.654s
2023-01-11T22:03:10.2205094Z 
2023-01-11T22:03:10.2205156Z OK
2023-01-11T22:03:10.2205161Z 
2023-01-11T22:03:10.2205247Z Generating XML reports...
2023-01-11T22:03:10.2205619Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214407.xml
2023-01-11T22:03:10.2205899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2206027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2206309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2206432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2206617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5a1018r
2023-01-11T22:03:10.2206810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5a1018r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2207068Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2207074Z 
2023-01-11T22:03:10.2207149Z Running tests...
2023-01-11T22:03:10.2207345Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2207573Z   test_disable_gil_profiling (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13734
2023-01-11T22:03:10.2207728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13735
2023-01-11T22:03:10.2207882Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13736
2023-01-11T22:03:10.2208022Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13737
2023-01-11T22:03:10.2208295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2208417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2208703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2208840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2209026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvdpylc2
2023-01-11T22:03:10.2209222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvdpylc2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2209495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2209605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2209887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2210022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2210210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsttb7xf4
2023-01-11T22:03:10.2210405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsttb7xf4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2210679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2210803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2211082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2211216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2211385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfuna22on
2023-01-11T22:03:10.2211579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfuna22on/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2211882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2212007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2212314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2212449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2212633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmy5lttx
2023-01-11T22:03:10.2212828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmy5lttx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2212979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2213143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2213306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2213471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2213539Z ok (2.649s)
2023-01-11T22:03:10.2213547Z 
2023-01-11T22:03:10.2213747Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2213822Z Ran 1 test in 2.649s
2023-01-11T22:03:10.2213827Z 
2023-01-11T22:03:10.2213891Z OK
2023-01-11T22:03:10.2213896Z 
2023-01-11T22:03:10.2213979Z Generating XML reports...
2023-01-11T22:03:10.2214305Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214414.xml
2023-01-11T22:03:10.2214580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2214704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2214982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2215120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2215308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprz8bparf
2023-01-11T22:03:10.2215506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprz8bparf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2215761Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2215767Z 
2023-01-11T22:03:10.2215843Z Running tests...
2023-01-11T22:03:10.2216025Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2216249Z   test_dist_init_decorator (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13957
2023-01-11T22:03:10.2216405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13958
2023-01-11T22:03:10.2216562Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13959
2023-01-11T22:03:10.2216715Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13960
2023-01-11T22:03:10.2216994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2217115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2217394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2217515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2217700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp144r8xcl
2023-01-11T22:03:10.2217893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp144r8xcl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2218162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2218320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2218629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2218766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2218951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoh8j53rh
2023-01-11T22:03:10.2219132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoh8j53rh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2219406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2219531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2219812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2219950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2220138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkoxgdj_
2023-01-11T22:03:10.2220333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkoxgdj_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2220606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2220731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2220998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2221137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2221321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_720vpo
2023-01-11T22:03:10.2221513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_720vpo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2221803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2221971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2222135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2222299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2222354Z ok (2.740s)
2023-01-11T22:03:10.2222373Z 
2023-01-11T22:03:10.2222559Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2222636Z Ran 1 test in 2.740s
2023-01-11T22:03:10.2222641Z 
2023-01-11T22:03:10.2222703Z OK
2023-01-11T22:03:10.2222709Z 
2023-01-11T22:03:10.2222796Z Generating XML reports...
2023-01-11T22:03:10.2223140Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214419.xml
2023-01-11T22:03:10.2223421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2223550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2223832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2223955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2224141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdll6pxqn
2023-01-11T22:03:10.2224337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdll6pxqn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2224591Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2224597Z 
2023-01-11T22:03:10.2224671Z Running tests...
2023-01-11T22:03:10.2224923Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2225182Z   test_duplicate_name (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14180
2023-01-11T22:03:10.2225344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14181
2023-01-11T22:03:10.2225501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14182
2023-01-11T22:03:10.2225644Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14183
2023-01-11T22:03:10.2225921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2226045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2226332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2226470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2226658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps5czm19o
2023-01-11T22:03:10.2226858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps5czm19o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2227129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2227238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2227520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2227659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2227844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnr5fj1w7
2023-01-11T22:03:10.2228036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnr5fj1w7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2228306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2228432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2228713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2228846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2229016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45jcee2x
2023-01-11T22:03:10.2229207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45jcee2x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2229478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2229600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2229878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2230015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2230202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo4bo1m_h
2023-01-11T22:03:10.2230397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo4bo1m_h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2230549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2230711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2230874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2231039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2231107Z ok (2.448s)
2023-01-11T22:03:10.2231113Z 
2023-01-11T22:03:10.2231343Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2231420Z Ran 1 test in 2.448s
2023-01-11T22:03:10.2231425Z 
2023-01-11T22:03:10.2231486Z OK
2023-01-11T22:03:10.2231492Z 
2023-01-11T22:03:10.2231623Z Generating XML reports...
2023-01-11T22:03:10.2231953Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214423.xml
2023-01-11T22:03:10.2232227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2232352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2232631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2232766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2232947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zkembm3
2023-01-11T22:03:10.2233146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zkembm3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2233405Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2233412Z 
2023-01-11T22:03:10.2233487Z Running tests...
2023-01-11T22:03:10.2233669Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2233889Z   test_duplicate_name_2 (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14315
2023-01-11T22:03:10.2234046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14316
2023-01-11T22:03:10.2234202Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14317
2023-01-11T22:03:10.2234357Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14318
2023-01-11T22:03:10.2234633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2234760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2235045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2235168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2235357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphw5xdfma
2023-01-11T22:03:10.2235558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphw5xdfma/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2235830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2235955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2236234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2236378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2236563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwiu8tq0
2023-01-11T22:03:10.2236743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwiu8tq0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2237012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2237135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2237413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2237548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2237731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwn45sqc8
2023-01-11T22:03:10.2237922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwn45sqc8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2238257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2238382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2238649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2238781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2239038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa64uymoy
2023-01-11T22:03:10.2239236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa64uymoy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2239401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2239567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2239734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2239898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2239952Z ok (2.436s)
2023-01-11T22:03:10.2239972Z 
2023-01-11T22:03:10.2240157Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2240234Z Ran 1 test in 2.436s
2023-01-11T22:03:10.2240239Z 
2023-01-11T22:03:10.2240302Z OK
2023-01-11T22:03:10.2240307Z 
2023-01-11T22:03:10.2240392Z Generating XML reports...
2023-01-11T22:03:10.2240730Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214428.xml
2023-01-11T22:03:10.2241005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2241131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2241417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2241541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2241727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzbg4ce4
2023-01-11T22:03:10.2241923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzbg4ce4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2242181Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2242187Z 
2023-01-11T22:03:10.2242261Z Running tests...
2023-01-11T22:03:10.2242456Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2242674Z   test_expected_src (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14450
2023-01-11T22:03:10.2242834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14451
2023-01-11T22:03:10.2242992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14452
2023-01-11T22:03:10.2243136Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14453
2023-01-11T22:03:10.2243409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2243532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2243813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2243948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2244132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_3u8y7b
2023-01-11T22:03:10.2244325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_3u8y7b/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2244633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2244772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2245053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2245188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2245371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8bve89jv
2023-01-11T22:03:10.2245565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8bve89jv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2245835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2245958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2246245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2246380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2246554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28hi594w
2023-01-11T22:03:10.2246747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28hi594w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2247022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2247144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2247428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2247563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2247747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3j6uer9r
2023-01-11T22:03:10.2247943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3j6uer9r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2248097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2248264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2248429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2248589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2248656Z ok (2.641s)
2023-01-11T22:03:10.2248662Z 
2023-01-11T22:03:10.2248859Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2248934Z Ran 1 test in 2.641s
2023-01-11T22:03:10.2248940Z 
2023-01-11T22:03:10.2249000Z OK
2023-01-11T22:03:10.2249004Z 
2023-01-11T22:03:10.2249087Z Generating XML reports...
2023-01-11T22:03:10.2249418Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214432.xml
2023-01-11T22:03:10.2249694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2249818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2250098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2250234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2250420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxe3ebe1
2023-01-11T22:03:10.2250616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxe3ebe1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2250870Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2250911Z 
2023-01-11T22:03:10.2250973Z Running tests...
2023-01-11T22:03:10.2251172Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2251425Z   test_function_not_on_callee (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14673
2023-01-11T22:03:10.2251585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14674
2023-01-11T22:03:10.2251740Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14675
2023-01-11T22:03:10.2251895Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14676
2023-01-11T22:03:10.2252171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2252296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2252575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2252699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2252889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9nlvjhz
2023-01-11T22:03:10.2253083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9nlvjhz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2253356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2253479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2253759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2253894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2254080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmnbfq9qj
2023-01-11T22:03:10.2254265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmnbfq9qj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2254535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2254658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2254938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2255071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2255254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8e3yno1m
2023-01-11T22:03:10.2255448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8e3yno1m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2255717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2255842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2256106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2256243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2256426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgiga45jr
2023-01-11T22:03:10.2256618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgiga45jr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2256782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2256944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2257108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2257269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2257383Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.2258071Z AttributeError("Can't get attribute 'foo_add' on <module 'torch.testing._internal.distributed.rpc.rpc_test' from '/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py'> Default RPC pickler does not serialize\n            function code. Ensure that UDFs are defined on both caller and\n            callee modules.")
2023-01-11T22:03:10.2258167Z Traceback (most recent call last):
2023-01-11T22:03:10.2258429Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 160, in deserialize
2023-01-11T22:03:10.2258515Z     ret = unpickler.load()
2023-01-11T22:03:10.2258947Z AttributeError: Can't get attribute 'foo_add' on <module 'torch.testing._internal.distributed.rpc.rpc_test' from '/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py'>
2023-01-11T22:03:10.2258955Z 
2023-01-11T22:03:10.2259097Z The above exception was the direct cause of the following exception:
2023-01-11T22:03:10.2259103Z 
2023-01-11T22:03:10.2259196Z Traceback (most recent call last):
2023-01-11T22:03:10.2259458Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2023-01-11T22:03:10.2259535Z     raise python_udf
2023-01-11T22:03:10.2260011Z AttributeError: Can't get attribute 'foo_add' on <module 'torch.testing._internal.distributed.rpc.rpc_test' from '/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py'> Default RPC pickler does not serialize
2023-01-11T22:03:10.2260146Z             function code. Ensure that UDFs are defined on both caller and
2023-01-11T22:03:10.2260223Z             callee modules.
2023-01-11T22:03:10.2260228Z 
2023-01-11T22:03:10.2260296Z ok (2.751s)
2023-01-11T22:03:10.2260301Z 
2023-01-11T22:03:10.2260495Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2260573Z Ran 1 test in 2.751s
2023-01-11T22:03:10.2260581Z 
2023-01-11T22:03:10.2260643Z OK
2023-01-11T22:03:10.2260647Z 
2023-01-11T22:03:10.2260729Z Generating XML reports...
2023-01-11T22:03:10.2261074Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214437.xml
2023-01-11T22:03:10.2261338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2261462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2261868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2262007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2262199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgcjvh_s
2023-01-11T22:03:10.2262402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgcjvh_s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2262660Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2262666Z 
2023-01-11T22:03:10.2262745Z Running tests...
2023-01-11T22:03:10.2262944Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2263154Z   test_future_done (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14896
2023-01-11T22:03:10.2263315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14897
2023-01-11T22:03:10.2263472Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14898
2023-01-11T22:03:10.2263628Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14899
2023-01-11T22:03:10.2263906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2264032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2264380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2264552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2264728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoiy87lca
2023-01-11T22:03:10.2264927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoiy87lca/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2265202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2265330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2265614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2265752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2265940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ysvddwv
2023-01-11T22:03:10.2266139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ysvddwv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2266413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2266526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2266807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2266943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2267129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu946ud0t
2023-01-11T22:03:10.2267324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu946ud0t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2267601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2267726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2268007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2268129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2268312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjp1g85yg
2023-01-11T22:03:10.2268507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjp1g85yg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2268672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2268837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2268999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2269163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2269231Z ok (2.642s)
2023-01-11T22:03:10.2269239Z 
2023-01-11T22:03:10.2269439Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2269501Z Ran 1 test in 2.642s
2023-01-11T22:03:10.2269506Z 
2023-01-11T22:03:10.2269568Z OK
2023-01-11T22:03:10.2269574Z 
2023-01-11T22:03:10.2269656Z Generating XML reports...
2023-01-11T22:03:10.2269999Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214441.xml
2023-01-11T22:03:10.2270275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2270398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2270681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2270850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2271052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpplza4ov8
2023-01-11T22:03:10.2271251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpplza4ov8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2271505Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2271511Z 
2023-01-11T22:03:10.2271583Z Running tests...
2023-01-11T22:03:10.2271780Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2272006Z   test_future_done_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15119
2023-01-11T22:03:10.2272161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15120
2023-01-11T22:03:10.2272317Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15121
2023-01-11T22:03:10.2272472Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15122
2023-01-11T22:03:10.2272738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2272861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2273147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2273285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2273471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13yff3oi
2023-01-11T22:03:10.2273668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13yff3oi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2273942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2274070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2274355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2274480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2274668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj845kegd
2023-01-11T22:03:10.2274865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj845kegd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2275138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2275263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2275544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2275683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2275871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5mghvz8d
2023-01-11T22:03:10.2276053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5mghvz8d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2276326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2276450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2276731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2276867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2277052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvoudzgup
2023-01-11T22:03:10.2277254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvoudzgup/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2277451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2277690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2277843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2278007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2278101Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.2278226Z ValueError('Expected error')
2023-01-11T22:03:10.2278320Z Traceback (most recent call last):
2023-01-11T22:03:10.2278584Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2278720Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2279092Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.2279177Z     raise ValueError(expected_err)
2023-01-11T22:03:10.2279265Z ValueError: Expected error
2023-01-11T22:03:10.2279272Z 
2023-01-11T22:03:10.2279366Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.2279490Z ValueError('Expected error')
2023-01-11T22:03:10.2279587Z Traceback (most recent call last):
2023-01-11T22:03:10.2279848Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2279981Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2280250Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.2280347Z     raise ValueError(expected_err)
2023-01-11T22:03:10.2280433Z ValueError: Expected error
2023-01-11T22:03:10.2280438Z 
2023-01-11T22:03:10.2280529Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.2280652Z ValueError('Expected error')
2023-01-11T22:03:10.2280744Z Traceback (most recent call last):
2023-01-11T22:03:10.2281002Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2281137Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2281404Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.2281499Z     raise ValueError(expected_err)
2023-01-11T22:03:10.2281585Z ValueError: Expected error
2023-01-11T22:03:10.2281590Z 
2023-01-11T22:03:10.2281680Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.2281800Z ValueError('Expected error')
2023-01-11T22:03:10.2281893Z Traceback (most recent call last):
2023-01-11T22:03:10.2282148Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2282283Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2282547Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.2282643Z     raise ValueError(expected_err)
2023-01-11T22:03:10.2282727Z ValueError: Expected error
2023-01-11T22:03:10.2282732Z 
2023-01-11T22:03:10.2282800Z ok (2.751s)
2023-01-11T22:03:10.2282806Z 
2023-01-11T22:03:10.2283006Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2283080Z Ran 1 test in 2.751s
2023-01-11T22:03:10.2283085Z 
2023-01-11T22:03:10.2283145Z OK
2023-01-11T22:03:10.2283152Z 
2023-01-11T22:03:10.2283235Z Generating XML reports...
2023-01-11T22:03:10.2283562Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214446.xml
2023-01-11T22:03:10.2283839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2284004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2284312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2284449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2284635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgftfyuy
2023-01-11T22:03:10.2284833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgftfyuy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2285088Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2285095Z 
2023-01-11T22:03:10.2285168Z Running tests...
2023-01-11T22:03:10.2285351Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2285569Z   test_future_in_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15342
2023-01-11T22:03:10.2285730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15343
2023-01-11T22:03:10.2285889Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15344
2023-01-11T22:03:10.2286046Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15345
2023-01-11T22:03:10.2286322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2286445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2286727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2286850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2287035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpym1vc3zs
2023-01-11T22:03:10.2287234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpym1vc3zs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2287510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2287637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2287920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2288055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2288241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlhf_qbq
2023-01-11T22:03:10.2288512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2288621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2288901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2289039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2289225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vvcsl18
2023-01-11T22:03:10.2289422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlhf_qbq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2289611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vvcsl18/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2289884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2290007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2290277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2290410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2290627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoz4nqqtn
2023-01-11T22:03:10.2290852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoz4nqqtn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2291016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2291180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2291343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2291505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2291571Z ok (2.637s)
2023-01-11T22:03:10.2291579Z 
2023-01-11T22:03:10.2291767Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2291842Z Ran 1 test in 2.637s
2023-01-11T22:03:10.2291847Z 
2023-01-11T22:03:10.2291913Z OK
2023-01-11T22:03:10.2291918Z 
2023-01-11T22:03:10.2292004Z Generating XML reports...
2023-01-11T22:03:10.2292350Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214450.xml
2023-01-11T22:03:10.2292625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2292753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2293032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2293155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2293339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwp54t0m
2023-01-11T22:03:10.2293535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwp54t0m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2293794Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2293800Z 
2023-01-11T22:03:10.2293877Z Running tests...
2023-01-11T22:03:10.2294075Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2294304Z   test_future_nested_callback (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15565
2023-01-11T22:03:10.2294463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15566
2023-01-11T22:03:10.2294618Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15567
2023-01-11T22:03:10.2294759Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15568
2023-01-11T22:03:10.2295032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2295156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2295439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2295576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2295763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmple6nk1j7
2023-01-11T22:03:10.2295957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmple6nk1j7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2296227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2296351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2296620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2296754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2296938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkjinprar
2023-01-11T22:03:10.2297165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkjinprar/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2297472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2297594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2297904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2298039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2298210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6p54wmt
2023-01-11T22:03:10.2298407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6p54wmt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2298676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2298801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2299081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2299215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2299399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1fxrym4
2023-01-11T22:03:10.2299592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1fxrym4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2299755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2299907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2300069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2300232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2300299Z ok (2.849s)
2023-01-11T22:03:10.2300305Z 
2023-01-11T22:03:10.2300508Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2300583Z Ran 1 test in 2.849s
2023-01-11T22:03:10.2300589Z 
2023-01-11T22:03:10.2300648Z OK
2023-01-11T22:03:10.2300653Z 
2023-01-11T22:03:10.2300737Z Generating XML reports...
2023-01-11T22:03:10.2301064Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214455.xml
2023-01-11T22:03:10.2301337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2301460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2301861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2302001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2302188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_q6bqjgn
2023-01-11T22:03:10.2302383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_q6bqjgn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2302639Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2302645Z 
2023-01-11T22:03:10.2302720Z Running tests...
2023-01-11T22:03:10.2302905Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2303585Z   test_future_wait_twice (__main__.TensorPipeRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/69480 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.314s)
2023-01-11T22:03:10.2303670Z 
2023-01-11T22:03:10.2303868Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2303945Z Ran 1 test in 0.314s
2023-01-11T22:03:10.2303991Z 
2023-01-11T22:03:10.2304068Z OK (skipped=1)
2023-01-11T22:03:10.2304073Z 
2023-01-11T22:03:10.2304161Z Generating XML reports...
2023-01-11T22:03:10.2304501Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214500.xml
2023-01-11T22:03:10.2304775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2304900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2305181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2305304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2305495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9dywl2yl
2023-01-11T22:03:10.2305691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9dywl2yl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2305945Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2305951Z 
2023-01-11T22:03:10.2306026Z Running tests...
2023-01-11T22:03:10.2306219Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2306440Z   test_get_worker_infos (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15798
2023-01-11T22:03:10.2306598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15799
2023-01-11T22:03:10.2306740Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15800
2023-01-11T22:03:10.2306895Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15801
2023-01-11T22:03:10.2307173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2307300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2307583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2307718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2307904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55a8msdz
2023-01-11T22:03:10.2308099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55a8msdz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2308374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2308485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2308769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2308904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2309090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxur0zc7g
2023-01-11T22:03:10.2309286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxur0zc7g/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2309556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2309680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2309959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2310082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2310267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9b408iw
2023-01-11T22:03:10.2310493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9b408iw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2310795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2310920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2311203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2311338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2311523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_ewqcdm
2023-01-11T22:03:10.2311717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_ewqcdm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2311869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2312034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2312200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2312362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2312430Z ok (2.649s)
2023-01-11T22:03:10.2312436Z 
2023-01-11T22:03:10.2312631Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2312707Z Ran 1 test in 2.650s
2023-01-11T22:03:10.2312713Z 
2023-01-11T22:03:10.2312774Z OK
2023-01-11T22:03:10.2312779Z 
2023-01-11T22:03:10.2312850Z Generating XML reports...
2023-01-11T22:03:10.2313192Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214502.xml
2023-01-11T22:03:10.2313471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2313599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2313880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2314015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2314200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcs4dj8l
2023-01-11T22:03:10.2314397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcs4dj8l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2314653Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2314658Z 
2023-01-11T22:03:10.2314718Z Running tests...
2023-01-11T22:03:10.2314915Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2315064Z   test_graceful_shutdown_with_uneven_workload (__main__.TensorPipeRpcTest)
2023-01-11T22:03:10.2315260Z Test graceful termination. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16021
2023-01-11T22:03:10.2315419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16022
2023-01-11T22:03:10.2315576Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16023
2023-01-11T22:03:10.2315731Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16024
2023-01-11T22:03:10.2316007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2316119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2316399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2316534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2316719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgkc2t6q
2023-01-11T22:03:10.2316947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgkc2t6q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2317509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2317837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2318279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2318631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2319080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2imnc5i_
2023-01-11T22:03:10.2319474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2imnc5i_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2319965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2320303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2320733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2321063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2321411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_uimb45
2023-01-11T22:03:10.2321807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_uimb45/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2322281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2322612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2323043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2323391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2323726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxi4ihyci
2023-01-11T22:03:10.2324134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxi4ihyci/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2324512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2324863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2325200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2325546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2325801Z ok (2.847s)
2023-01-11T22:03:10.2325905Z 
2023-01-11T22:03:10.2326092Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2326344Z Ran 1 test in 2.847s
2023-01-11T22:03:10.2326456Z 
2023-01-11T22:03:10.2326519Z OK
2023-01-11T22:03:10.2326610Z 
2023-01-11T22:03:10.2326682Z Generating XML reports...
2023-01-11T22:03:10.2327147Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214507.xml
2023-01-11T22:03:10.2327689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2328022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2328439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2328784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2329127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ol88vzq
2023-01-11T22:03:10.2329527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ol88vzq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2330056Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2330267Z 
2023-01-11T22:03:10.2330390Z Running tests...
2023-01-11T22:03:10.2330703Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2331091Z   test_handle_send_exceptions (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16244
2023-01-11T22:03:10.2331488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16245
2023-01-11T22:03:10.2331825Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16246
2023-01-11T22:03:10.2332162Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16247
2023-01-11T22:03:10.2332602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2332951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2333407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2333754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2334099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1iom1djr
2023-01-11T22:03:10.2334494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1iom1djr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2334981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2335317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2335734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2336077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2336424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4k0fel17
2023-01-11T22:03:10.2336836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4k0fel17/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2337310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2337645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2338073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2338407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2338750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0izhlnk
2023-01-11T22:03:10.2339154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0izhlnk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2339648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2339971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2340432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2340803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2341154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphr087486
2023-01-11T22:03:10.2341540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphr087486/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2342041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2342403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2342753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2343325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2350783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.2351186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.2351548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.2351924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.2352474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2353010Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2353523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2354052Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2354552Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.2355107Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.2355638Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.2356167Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.2356730Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.2357295Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.2357822Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.2358366Z [W tensorpipe_agent.cpp:915] RPC agent for worker1 encountered error when sending outgoing request #1 to worker2: EOF: end of file (this error originated at tensorpipe/transport/uv/connection_impl.cc:132)
2023-01-11T22:03:10.2359012Z [W tensorpipe_agent.cpp:915] RPC agent for worker1 encountered error when sending outgoing request #2 to worker2: EOF: end of file (this error originated at tensorpipe/transport/uv/connection_impl.cc:132)
2023-01-11T22:03:10.2359366Z ok (2.745s)
2023-01-11T22:03:10.2359478Z 
2023-01-11T22:03:10.2359689Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2359943Z Ran 1 test in 2.746s
2023-01-11T22:03:10.2360043Z 
2023-01-11T22:03:10.2360106Z OK
2023-01-11T22:03:10.2360201Z 
2023-01-11T22:03:10.2360290Z Generating XML reports...
2023-01-11T22:03:10.2360759Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214511.xml
2023-01-11T22:03:10.2361292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2361628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2362131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2362518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2362867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojym9mpx
2023-01-11T22:03:10.2363324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojym9mpx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2363833Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2364049Z 
2023-01-11T22:03:10.2364138Z Running tests...
2023-01-11T22:03:10.2364451Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2364845Z   test_ignore_rref_leak (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16479
2023-01-11T22:03:10.2365242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16480
2023-01-11T22:03:10.2365570Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16481
2023-01-11T22:03:10.2365937Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16482
2023-01-11T22:03:10.2366409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2366746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2367178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2367542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2367899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsm2xgnhn
2023-01-11T22:03:10.2368319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsm2xgnhn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2368819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2369157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2369601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2369950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2370282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbrper4hy
2023-01-11T22:03:10.2370695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbrper4hy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2371187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2371515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2371948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2372303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2372647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumtzlxxx
2023-01-11T22:03:10.2373036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumtzlxxx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2373532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2373867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2374297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2374629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2374977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbpqz2cp
2023-01-11T22:03:10.2375429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbpqz2cp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2375828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2376181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2376528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2376876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2377236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.2377612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.2377989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.2378354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.2378872Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2379409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2379932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2380443Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2381223Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2023-01-11T22:03:10.2382247Z Leaking RRef GloballyUniqueId(created_on=3, local_id=0) with fork Id GloballyUniqueId(created_on=3, local_id=1)
2023-01-11T22:03:10.2382461Z 
2023-01-11T22:03:10.2383011Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2023-01-11T22:03:10.2383741Z Leaking RRef GloballyUniqueId(created_on=1, local_id=0) with fork Id GloballyUniqueId(created_on=1, local_id=1)
2023-01-11T22:03:10.2383958Z 
2023-01-11T22:03:10.2384505Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2023-01-11T22:03:10.2385340Z Leaking RRef GloballyUniqueId(created_on=0, local_id=0) with fork Id GloballyUniqueId(created_on=0, local_id=1)
2023-01-11T22:03:10.2385551Z 
2023-01-11T22:03:10.2386185Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2023-01-11T22:03:10.2386948Z Leaking RRef GloballyUniqueId(created_on=2, local_id=0) with fork Id GloballyUniqueId(created_on=2, local_id=1)
2023-01-11T22:03:10.2387145Z 
2023-01-11T22:03:10.2387217Z ok (2.745s)
2023-01-11T22:03:10.2387325Z 
2023-01-11T22:03:10.2387540Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2387793Z Ran 1 test in 2.745s
2023-01-11T22:03:10.2387909Z 
2023-01-11T22:03:10.2387958Z OK
2023-01-11T22:03:10.2388051Z 
2023-01-11T22:03:10.2388138Z Generating XML reports...
2023-01-11T22:03:10.2388607Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214516.xml
2023-01-11T22:03:10.2389147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2389471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2389908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2390261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2390596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgjk0omv
2023-01-11T22:03:10.2391006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgjk0omv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2391480Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2391691Z 
2023-01-11T22:03:10.2391766Z Running tests...
2023-01-11T22:03:10.2392057Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2392443Z   test_init_pg_then_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16714
2023-01-11T22:03:10.2392836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16715
2023-01-11T22:03:10.2393172Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16716
2023-01-11T22:03:10.2393494Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16717
2023-01-11T22:03:10.2393948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2394283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2394703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2395052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2395399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvn399yae
2023-01-11T22:03:10.2395878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvn399yae/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2396393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2396729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2397164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2397501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2397877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptq49c1x5
2023-01-11T22:03:10.2398280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptq49c1x5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2398770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2399217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2399691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2400041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2400386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnuty5dq
2023-01-11T22:03:10.2400774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnuty5dq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2401268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2401600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2402016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2402361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2402707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplyjo2skc
2023-01-11T22:03:10.2403115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplyjo2skc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2403479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2403832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2404183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2404519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2404880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.2405254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.2405756Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2406141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.2406636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2407158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2407559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.2408035Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2408321Z ok (2.750s)
2023-01-11T22:03:10.2408424Z 
2023-01-11T22:03:10.2408624Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2408875Z Ran 1 test in 2.751s
2023-01-11T22:03:10.2408975Z 
2023-01-11T22:03:10.2409038Z OK
2023-01-11T22:03:10.2409132Z 
2023-01-11T22:03:10.2409222Z Generating XML reports...
2023-01-11T22:03:10.2409686Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214521.xml
2023-01-11T22:03:10.2410209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2410546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2410980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2411333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2411668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn23aedva
2023-01-11T22:03:10.2412119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn23aedva/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2412625Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2412836Z 
2023-01-11T22:03:10.2412897Z Running tests...
2023-01-11T22:03:10.2413204Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2413595Z   test_init_rpc_then_pg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16949
2023-01-11T22:03:10.2413987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16950
2023-01-11T22:03:10.2414311Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16951
2023-01-11T22:03:10.2414641Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16952
2023-01-11T22:03:10.2415096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2415418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2415854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2416202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2416547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78x75j34
2023-01-11T22:03:10.2416935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78x75j34/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2417482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2417862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2418298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2418636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2418983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv41qv82i
2023-01-11T22:03:10.2419388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv41qv82i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2419961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2420294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2420818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2421166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2421497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnid_x08l
2023-01-11T22:03:10.2422012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnid_x08l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2422509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2422829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2423262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2423614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2423959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1jfrc0s
2023-01-11T22:03:10.2424344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1jfrc0s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2424723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2425079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2425515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2425911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2426271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.2426649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.2427005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.2427376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.2427876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2428578Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2429104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2429625Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2429913Z ok (2.744s)
2023-01-11T22:03:10.2430017Z 
2023-01-11T22:03:10.2430221Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2430452Z Ran 1 test in 2.744s
2023-01-11T22:03:10.2430568Z 
2023-01-11T22:03:10.2430631Z OK
2023-01-11T22:03:10.2430726Z 
2023-01-11T22:03:10.2430816Z Generating XML reports...
2023-01-11T22:03:10.2431268Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214525.xml
2023-01-11T22:03:10.2431802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2432143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2432576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2432913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2433261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa15i1gtc
2023-01-11T22:03:10.2433669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa15i1gtc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2434124Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2434331Z 
2023-01-11T22:03:10.2434406Z Running tests...
2023-01-11T22:03:10.2434711Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2435100Z   test_init_rpc_twice (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17184
2023-01-11T22:03:10.2435484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17185
2023-01-11T22:03:10.2435824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17186
2023-01-11T22:03:10.2436156Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17187
2023-01-11T22:03:10.2436613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2436935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2437369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2437717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2438050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8k7jyzzj
2023-01-11T22:03:10.2438455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8k7jyzzj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2439070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2439437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2439858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2440205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2440557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4_byw1q
2023-01-11T22:03:10.2440946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4_byw1q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2441437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2441768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2442204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2442538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2442888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg3nl3e8m
2023-01-11T22:03:10.2443296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg3nl3e8m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2443786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2444102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2444535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2444877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2445208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5m6qlzn
2023-01-11T22:03:10.2445613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5m6qlzn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2445991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2446343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2446680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2447019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2447382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.2447739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.2448242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2448645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.2449006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.2449486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2450008Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2450526Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2450810Z ok (2.937s)
2023-01-11T22:03:10.2450899Z 
2023-01-11T22:03:10.2451100Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2451417Z Ran 1 test in 2.938s
2023-01-11T22:03:10.2451531Z 
2023-01-11T22:03:10.2451596Z OK
2023-01-11T22:03:10.2451689Z 
2023-01-11T22:03:10.2451764Z Generating XML reports...
2023-01-11T22:03:10.2452259Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214530.xml
2023-01-11T22:03:10.2452806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2453146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2453572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2453924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2454272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyra41jr
2023-01-11T22:03:10.2454661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyra41jr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2455135Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2455343Z 
2023-01-11T22:03:10.2455420Z Running tests...
2023-01-11T22:03:10.2455724Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2456097Z   test_int_callee (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17583
2023-01-11T22:03:10.2456482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17584
2023-01-11T22:03:10.2456815Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17585
2023-01-11T22:03:10.2457148Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17586
2023-01-11T22:03:10.2457591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2457925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2458360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2458697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2459047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnxy5yd3q
2023-01-11T22:03:10.2459454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnxy5yd3q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2459944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2460265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2460700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2461048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2461381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocy0t8bs
2023-01-11T22:03:10.2461987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2462318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2462747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2463084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2463430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7fi7wbj
2023-01-11T22:03:10.2463835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocy0t8bs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2464247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7fi7wbj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2464831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2465171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2465646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2465984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2466334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtz3qgdh
2023-01-11T22:03:10.2466737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtz3qgdh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2467116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2467457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2467804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2468156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2468394Z ok (2.638s)
2023-01-11T22:03:10.2468502Z 
2023-01-11T22:03:10.2468705Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2468949Z Ran 1 test in 2.638s
2023-01-11T22:03:10.2469062Z 
2023-01-11T22:03:10.2469124Z OK
2023-01-11T22:03:10.2469204Z 
2023-01-11T22:03:10.2469290Z Generating XML reports...
2023-01-11T22:03:10.2469748Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214535.xml
2023-01-11T22:03:10.2470286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2470605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2471036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2471386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2471737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0uyn1wlv
2023-01-11T22:03:10.2472129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0uyn1wlv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2472604Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2472815Z 
2023-01-11T22:03:10.2472890Z Running tests...
2023-01-11T22:03:10.2473195Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2473565Z   test_invalid_names (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17806
2023-01-11T22:03:10.2473952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17807
2023-01-11T22:03:10.2474291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17808
2023-01-11T22:03:10.2474609Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17809
2023-01-11T22:03:10.2475063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2475396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2475830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2476165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2476510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpot55w0ld
2023-01-11T22:03:10.2476917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpot55w0ld/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2477394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2477758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2478223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2478573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2478977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsk6nq3fc
2023-01-11T22:03:10.2479388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsk6nq3fc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2480002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2480336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2480755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2481106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2481456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpztp4uu0y
2023-01-11T22:03:10.2481847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpztp4uu0y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2482339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2482669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2483103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2483438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2483782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3kgktoss
2023-01-11T22:03:10.2484188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3kgktoss/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2484555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2484906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2485254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2485605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2485847Z ok (2.445s)
2023-01-11T22:03:10.2485951Z 
2023-01-11T22:03:10.2486157Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2486404Z Ran 1 test in 2.445s
2023-01-11T22:03:10.2486518Z 
2023-01-11T22:03:10.2486566Z OK
2023-01-11T22:03:10.2486660Z 
2023-01-11T22:03:10.2486747Z Generating XML reports...
2023-01-11T22:03:10.2487210Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214540.xml
2023-01-11T22:03:10.2487751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2488074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2488508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2488860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2489193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppk92w31e
2023-01-11T22:03:10.2489600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppk92w31e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2490070Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2490278Z 
2023-01-11T22:03:10.2490353Z Running tests...
2023-01-11T22:03:10.2490699Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2491122Z   test_local_rref_no_fork (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17865
2023-01-11T22:03:10.2491520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17866
2023-01-11T22:03:10.2491931Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17867
2023-01-11T22:03:10.2492250Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17868
2023-01-11T22:03:10.2492703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2493040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2493463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2493816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2494171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5s6_yp3
2023-01-11T22:03:10.2494580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5s6_yp3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2495061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2495398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2495831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2496182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2496514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitnrmsz6
2023-01-11T22:03:10.2496921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitnrmsz6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2497416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2497736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2498172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2498519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2498863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7bkjsslv
2023-01-11T22:03:10.2499253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7bkjsslv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2499751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2500089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2500507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2500853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2501208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpul0sg2jc
2023-01-11T22:03:10.2501608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpul0sg2jc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2502088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2502446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2502798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2503143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2503385Z ok (2.652s)
2023-01-11T22:03:10.2503489Z 
2023-01-11T22:03:10.2503775Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2504028Z Ran 1 test in 2.652s
2023-01-11T22:03:10.2504141Z 
2023-01-11T22:03:10.2504190Z OK
2023-01-11T22:03:10.2504285Z 
2023-01-11T22:03:10.2504413Z Generating XML reports...
2023-01-11T22:03:10.2504884Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214544.xml
2023-01-11T22:03:10.2505430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2505750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2506185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2506540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2506875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1zfjcaj0
2023-01-11T22:03:10.2507290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1zfjcaj0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2507766Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2507975Z 
2023-01-11T22:03:10.2508050Z Running tests...
2023-01-11T22:03:10.2508342Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2508731Z   test_local_shutdown (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18088
2023-01-11T22:03:10.2509121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18089
2023-01-11T22:03:10.2509459Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18090
2023-01-11T22:03:10.2509773Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18091
2023-01-11T22:03:10.2510229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2510566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2510993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2511364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2511714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaw0t542c
2023-01-11T22:03:10.2512120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaw0t542c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2512600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2512938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2513376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2513731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2514064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpob6ysllk
2023-01-11T22:03:10.2514473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpob6ysllk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2514974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2515299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2515729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2516078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2516429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz1u94p_h
2023-01-11T22:03:10.2516819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz1u94p_h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2517405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2517743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2518179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2518521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2518939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnrg3kq6w
2023-01-11T22:03:10.2519357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnrg3kq6w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2519729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2520090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2520514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2520709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2520987Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.2521262Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.2521528Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.2521779Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.2522043Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.2522303Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.2522376Z ok (2.752s)
2023-01-11T22:03:10.2522383Z 
2023-01-11T22:03:10.2522589Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2522667Z Ran 1 test in 2.753s
2023-01-11T22:03:10.2522673Z 
2023-01-11T22:03:10.2522738Z OK
2023-01-11T22:03:10.2522743Z 
2023-01-11T22:03:10.2522828Z Generating XML reports...
2023-01-11T22:03:10.2523176Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214548.xml
2023-01-11T22:03:10.2523458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2523574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2523861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2523999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2524185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzss_o3i1
2023-01-11T22:03:10.2524384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzss_o3i1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2524638Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2524644Z 
2023-01-11T22:03:10.2524726Z Running tests...
2023-01-11T22:03:10.2524969Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2525219Z   test_local_shutdown_with_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18311
2023-01-11T22:03:10.2525384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18312
2023-01-11T22:03:10.2525543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18313
2023-01-11T22:03:10.2525701Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18314
2023-01-11T22:03:10.2525979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2526105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2526395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2526532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2526725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_caiajov
2023-01-11T22:03:10.2526911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_caiajov/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2527189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2527319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2527602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2527739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2527927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09pymh2p
2023-01-11T22:03:10.2528127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09pymh2p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2528405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2528533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2528801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2528939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2529126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkq1ss6if
2023-01-11T22:03:10.2529402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2529531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2529820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2529958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2530144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gy1k4yk
2023-01-11T22:03:10.2530329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkq1ss6if/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2530522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gy1k4yk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2530692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2530856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2531021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2531184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2531365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.2531573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.2531778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.2531940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.2532246Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2532551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2532846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2533144Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2533422Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.2533689Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.2533957Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.2534219Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.2534475Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.2534734Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.2534995Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.2535240Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.2535307Z ok (2.741s)
2023-01-11T22:03:10.2535313Z 
2023-01-11T22:03:10.2535518Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2535597Z Ran 1 test in 2.742s
2023-01-11T22:03:10.2535603Z 
2023-01-11T22:03:10.2535667Z OK
2023-01-11T22:03:10.2535673Z 
2023-01-11T22:03:10.2535766Z Generating XML reports...
2023-01-11T22:03:10.2536112Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214553.xml
2023-01-11T22:03:10.2536392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2536519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2536791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2536927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2537117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48pzs67q
2023-01-11T22:03:10.2537318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48pzs67q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2537582Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2537624Z 
2023-01-11T22:03:10.2537698Z Running tests...
2023-01-11T22:03:10.2537926Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2538159Z   test_local_value_not_on_owner (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18546
2023-01-11T22:03:10.2538318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18547
2023-01-11T22:03:10.2538464Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18548
2023-01-11T22:03:10.2538621Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18549
2023-01-11T22:03:10.2538898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2539027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2539322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2539459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2539644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplrnoz_2o
2023-01-11T22:03:10.2539843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplrnoz_2o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2540104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2540230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2540511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2540650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2540833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6y7tavd2
2023-01-11T22:03:10.2541033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6y7tavd2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2541309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2541434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2541825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2541948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2542132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv57wj6zk
2023-01-11T22:03:10.2542330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv57wj6zk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2542602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2542735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2543020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2543158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2543343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnaczxlf
2023-01-11T22:03:10.2543537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnaczxlf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2543689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2543853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2544018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2544244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2544313Z ok (2.755s)
2023-01-11T22:03:10.2544320Z 
2023-01-11T22:03:10.2544558Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2544640Z Ran 1 test in 2.755s
2023-01-11T22:03:10.2544647Z 
2023-01-11T22:03:10.2544709Z OK
2023-01-11T22:03:10.2544715Z 
2023-01-11T22:03:10.2544787Z Generating XML reports...
2023-01-11T22:03:10.2545131Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214558.xml
2023-01-11T22:03:10.2545409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2545534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2545819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2545956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2546140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqp6keub
2023-01-11T22:03:10.2546339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqp6keub/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2546593Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2546599Z 
2023-01-11T22:03:10.2546659Z Running tests...
2023-01-11T22:03:10.2546861Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2547082Z   test_mark_future_twice (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18769
2023-01-11T22:03:10.2547238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18770
2023-01-11T22:03:10.2547398Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18771
2023-01-11T22:03:10.2547553Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18772
2023-01-11T22:03:10.2547827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2547954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2548236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2548357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2548544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjb5sc5a0
2023-01-11T22:03:10.2548738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjb5sc5a0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2549011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2549139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2549427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2549561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2549744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpomc28nam
2023-01-11T22:03:10.2549928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpomc28nam/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2550205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2550328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2550613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2550747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2550971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmp_bxux
2023-01-11T22:03:10.2551194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmp_bxux/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2551470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2551597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2551866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2552002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2552190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbzs0khle
2023-01-11T22:03:10.2552383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbzs0khle/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2552555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2552719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2552880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2553040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2553106Z ok (2.647s)
2023-01-11T22:03:10.2553112Z 
2023-01-11T22:03:10.2553297Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2553379Z Ran 1 test in 2.647s
2023-01-11T22:03:10.2553384Z 
2023-01-11T22:03:10.2553445Z OK
2023-01-11T22:03:10.2553451Z 
2023-01-11T22:03:10.2553534Z Generating XML reports...
2023-01-11T22:03:10.2553876Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214603.xml
2023-01-11T22:03:10.2554152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2554276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2554563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2554685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2554870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppckda0d6
2023-01-11T22:03:10.2555069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppckda0d6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2555330Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2555336Z 
2023-01-11T22:03:10.2555410Z Running tests...
2023-01-11T22:03:10.2555607Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2555842Z   test_multi_builtin_remote_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18992
2023-01-11T22:03:10.2556001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18993
2023-01-11T22:03:10.2556155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18994
2023-01-11T22:03:10.2556298Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18995
2023-01-11T22:03:10.2556573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2556695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2556981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2557118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2557344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4jk84e5z
2023-01-11T22:03:10.2557564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4jk84e5z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2557846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2557969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2558236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2558373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2558555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvxvu_ga
2023-01-11T22:03:10.2558753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvxvu_ga/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2559097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2559224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2559518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2559651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2559825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahhvc9oa
2023-01-11T22:03:10.2560022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahhvc9oa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2560300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2560424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2560713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2560851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2561045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprc2sdu07
2023-01-11T22:03:10.2561242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprc2sdu07/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2561412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2561563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2561725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2561889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2561957Z ok (2.646s)
2023-01-11T22:03:10.2561962Z 
2023-01-11T22:03:10.2562161Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2562239Z Ran 1 test in 2.646s
2023-01-11T22:03:10.2562245Z 
2023-01-11T22:03:10.2562305Z OK
2023-01-11T22:03:10.2562310Z 
2023-01-11T22:03:10.2562393Z Generating XML reports...
2023-01-11T22:03:10.2562725Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214607.xml
2023-01-11T22:03:10.2563002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2563130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2563412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2563548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2563738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkzojqdb_
2023-01-11T22:03:10.2563935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkzojqdb_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2564227Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2564278Z 
2023-01-11T22:03:10.2564355Z Running tests...
2023-01-11T22:03:10.2564542Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2564777Z   test_multi_layer_nested_async_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19215
2023-01-11T22:03:10.2564937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19216
2023-01-11T22:03:10.2565091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19217
2023-01-11T22:03:10.2565244Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19218
2023-01-11T22:03:10.2565521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2565647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2565935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2566070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2566243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdc_bdop
2023-01-11T22:03:10.2566438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdc_bdop/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2566719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2566843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2567134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2567271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2567459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp32n97y5
2023-01-11T22:03:10.2567655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp32n97y5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2567931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2568045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2568325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2568462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2568648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgxfes45l
2023-01-11T22:03:10.2568847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgxfes45l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2569125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2569250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2569536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2569676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2569850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwyt8fxn
2023-01-11T22:03:10.2570042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwyt8fxn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2570209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2570371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2570567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2570756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2570827Z ok (2.652s)
2023-01-11T22:03:10.2570833Z 
2023-01-11T22:03:10.2571030Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2571094Z Ran 1 test in 2.652s
2023-01-11T22:03:10.2571099Z 
2023-01-11T22:03:10.2571160Z OK
2023-01-11T22:03:10.2571165Z 
2023-01-11T22:03:10.2571256Z Generating XML reports...
2023-01-11T22:03:10.2571599Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214612.xml
2023-01-11T22:03:10.2571875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2572000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2572281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2572423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2572610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8c7i26l
2023-01-11T22:03:10.2572793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8c7i26l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2573051Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2573056Z 
2023-01-11T22:03:10.2573128Z Running tests...
2023-01-11T22:03:10.2573329Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2573551Z   test_multi_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19438
2023-01-11T22:03:10.2573708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19439
2023-01-11T22:03:10.2573869Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19440
2023-01-11T22:03:10.2574025Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19441
2023-01-11T22:03:10.2574286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2574415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2574695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2574835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2575022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp00pg9j75
2023-01-11T22:03:10.2575218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp00pg9j75/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2575495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2575622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2575907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2576030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2576217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxu5f6ts
2023-01-11T22:03:10.2576410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxu5f6ts/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2576687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2576811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2577089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2577254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2577468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpset56w0f
2023-01-11T22:03:10.2577663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpset56w0f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2577924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2578050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2578335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2578470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2578654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjj79a2mr
2023-01-11T22:03:10.2578851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjj79a2mr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2579021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2579185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2579335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2579499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2579567Z ok (2.747s)
2023-01-11T22:03:10.2579573Z 
2023-01-11T22:03:10.2579778Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2579853Z Ran 1 test in 2.747s
2023-01-11T22:03:10.2579859Z 
2023-01-11T22:03:10.2579921Z OK
2023-01-11T22:03:10.2579926Z 
2023-01-11T22:03:10.2580010Z Generating XML reports...
2023-01-11T22:03:10.2580352Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214616.xml
2023-01-11T22:03:10.2580633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2580746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2581023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2581159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2581342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8do1r4e
2023-01-11T22:03:10.2581545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8do1r4e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2581932Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2581939Z 
2023-01-11T22:03:10.2582016Z Running tests...
2023-01-11T22:03:10.2582221Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2582435Z   test_multi_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19661
2023-01-11T22:03:10.2582581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19662
2023-01-11T22:03:10.2582736Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19663
2023-01-11T22:03:10.2582890Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19664
2023-01-11T22:03:10.2583170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2583294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2583585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2583776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2583973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0l2l2lq
2023-01-11T22:03:10.2584228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0l2l2lq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2584503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2584631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2584909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2585049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2585232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61cxesq7
2023-01-11T22:03:10.2585433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61cxesq7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2585709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2585835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2586106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2586245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2586427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83jlhkw9
2023-01-11T22:03:10.2586626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83jlhkw9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2586903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2587031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2587313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2587449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2587621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5iks8ls
2023-01-11T22:03:10.2587816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5iks8ls/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2587983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2588149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2588315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2588475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2588547Z ok (2.650s)
2023-01-11T22:03:10.2588553Z 
2023-01-11T22:03:10.2588753Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2588829Z Ran 1 test in 2.650s
2023-01-11T22:03:10.2588835Z 
2023-01-11T22:03:10.2588885Z OK
2023-01-11T22:03:10.2588890Z 
2023-01-11T22:03:10.2588974Z Generating XML reports...
2023-01-11T22:03:10.2589317Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214621.xml
2023-01-11T22:03:10.2589594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2589723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2590002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2590138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2590323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsngk5uta
2023-01-11T22:03:10.2590561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsngk5uta/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2590831Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2590838Z 
2023-01-11T22:03:10.2590913Z Running tests...
2023-01-11T22:03:10.2591113Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2591340Z   test_my_parameter_server (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19884
2023-01-11T22:03:10.2591501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19885
2023-01-11T22:03:10.2591657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19886
2023-01-11T22:03:10.2591811Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19887
2023-01-11T22:03:10.2592095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2592210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2592497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2592635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2592820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptfg368t9
2023-01-11T22:03:10.2593013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptfg368t9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2593289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2593417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2593697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2593835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2594007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_zybmrfl
2023-01-11T22:03:10.2594205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_zybmrfl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2594477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2594601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2594881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2595019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2595205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7s04rwu
2023-01-11T22:03:10.2595400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7s04rwu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2595676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2595788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2596071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2596205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2596389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkszulm2_
2023-01-11T22:03:10.2596584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkszulm2_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2596751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2596945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2597135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2597285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2597357Z ok (2.850s)
2023-01-11T22:03:10.2597362Z 
2023-01-11T22:03:10.2597561Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2597637Z Ran 1 test in 2.850s
2023-01-11T22:03:10.2597642Z 
2023-01-11T22:03:10.2597706Z OK
2023-01-11T22:03:10.2597711Z 
2023-01-11T22:03:10.2597794Z Generating XML reports...
2023-01-11T22:03:10.2598134Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214625.xml
2023-01-11T22:03:10.2598411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2598539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2598810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2599039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2599226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05nuktxp
2023-01-11T22:03:10.2599424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05nuktxp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2599685Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2599691Z 
2023-01-11T22:03:10.2599764Z Running tests...
2023-01-11T22:03:10.2599967Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2600186Z   test_nested_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20143
2023-01-11T22:03:10.2600346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20144
2023-01-11T22:03:10.2600490Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20145
2023-01-11T22:03:10.2600649Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20146
2023-01-11T22:03:10.2600928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2601057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2601344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2601480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2601669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpneb2hhoa
2023-01-11T22:03:10.2601873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpneb2hhoa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2602136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2602262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2602543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2602679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2602864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8iwiafuh
2023-01-11T22:03:10.2603061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8iwiafuh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2603333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2603459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2603778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2603931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2604120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv14gwn1g
2023-01-11T22:03:10.2604317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv14gwn1g/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2604591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2604716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2604996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2605134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2605321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkm2umlt
2023-01-11T22:03:10.2605506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkm2umlt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2605672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2605837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2606001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2606163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2606233Z ok (2.555s)
2023-01-11T22:03:10.2606239Z 
2023-01-11T22:03:10.2606443Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2606507Z Ran 1 test in 2.555s
2023-01-11T22:03:10.2606513Z 
2023-01-11T22:03:10.2606575Z OK
2023-01-11T22:03:10.2606582Z 
2023-01-11T22:03:10.2606668Z Generating XML reports...
2023-01-11T22:03:10.2607014Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214630.xml
2023-01-11T22:03:10.2607287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2607412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2607692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2607829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2608016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8mkme26
2023-01-11T22:03:10.2608200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8mkme26/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2608457Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2608466Z 
2023-01-11T22:03:10.2608541Z Running tests...
2023-01-11T22:03:10.2608745Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2608968Z   test_nested_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20366
2023-01-11T22:03:10.2609126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20367
2023-01-11T22:03:10.2609286Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20368
2023-01-11T22:03:10.2609442Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20369
2023-01-11T22:03:10.2609703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2609832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2610119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2610288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2610526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_fwfsaf
2023-01-11T22:03:10.2610729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_fwfsaf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2611004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2611129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2611410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2611533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2611721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytmag0bf
2023-01-11T22:03:10.2611921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytmag0bf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2612196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2612321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2612600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2612737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2612923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvapm3w0i
2023-01-11T22:03:10.2613106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvapm3w0i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2613380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2613508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2613794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2613932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2614118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3muhvl6
2023-01-11T22:03:10.2614313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3muhvl6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2614479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2614642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2614793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2614958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2615031Z ok (2.749s)
2023-01-11T22:03:10.2615037Z 
2023-01-11T22:03:10.2615241Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2615318Z Ran 1 test in 2.749s
2023-01-11T22:03:10.2615324Z 
2023-01-11T22:03:10.2615388Z OK
2023-01-11T22:03:10.2615393Z 
2023-01-11T22:03:10.2615479Z Generating XML reports...
2023-01-11T22:03:10.2615820Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214635.xml
2023-01-11T22:03:10.2616081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2616209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2616490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2616674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2616863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3g_0xi2h
2023-01-11T22:03:10.2617086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3g_0xi2h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2617340Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2617346Z 
2023-01-11T22:03:10.2617419Z Running tests...
2023-01-11T22:03:10.2617616Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2617818Z   test_nested_rref (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20589
2023-01-11T22:03:10.2617974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20590
2023-01-11T22:03:10.2618130Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20591
2023-01-11T22:03:10.2618285Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20592
2023-01-11T22:03:10.2618565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2618688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2618971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2619105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2619279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4g0ui3l
2023-01-11T22:03:10.2619471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4g0ui3l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2619745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2619870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2620151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2620289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2620474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkowv2qvx
2023-01-11T22:03:10.2620669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkowv2qvx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2620988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2621102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2621381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2621539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2621873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0lk_oq7
2023-01-11T22:03:10.2622068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0lk_oq7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2622340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2622465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2622746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2622879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2623050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgpp6_wt
2023-01-11T22:03:10.2623245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgpp6_wt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2623470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2623636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2623831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2623995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2624065Z ok (2.639s)
2023-01-11T22:03:10.2624073Z 
2023-01-11T22:03:10.2624272Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2624335Z Ran 1 test in 2.640s
2023-01-11T22:03:10.2624341Z 
2023-01-11T22:03:10.2624403Z OK
2023-01-11T22:03:10.2624408Z 
2023-01-11T22:03:10.2624493Z Generating XML reports...
2023-01-11T22:03:10.2624833Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214639.xml
2023-01-11T22:03:10.2625108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2625236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2625517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2625653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2625835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8q9v07rr
2023-01-11T22:03:10.2626018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8q9v07rr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2626271Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2626277Z 
2023-01-11T22:03:10.2626350Z Running tests...
2023-01-11T22:03:10.2626548Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2626775Z   test_nested_rref_stress (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20812
2023-01-11T22:03:10.2626936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20813
2023-01-11T22:03:10.2627091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20814
2023-01-11T22:03:10.2627245Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20815
2023-01-11T22:03:10.2627507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2627629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2627908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2628041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2628228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mmzy6s_
2023-01-11T22:03:10.2628424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mmzy6s_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2628696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2628822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2629100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2629222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2629406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hq2taso
2023-01-11T22:03:10.2629599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hq2taso/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2629870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2630026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2630342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2630477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2630660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkjd0b9i8
2023-01-11T22:03:10.2630841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkjd0b9i8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2631114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2631237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2631516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2631651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2631833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0w4ryibb
2023-01-11T22:03:10.2632027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0w4ryibb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2632192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2632355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2632503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2632663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2632731Z ok (2.736s)
2023-01-11T22:03:10.2632737Z 
2023-01-11T22:03:10.2632933Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2633012Z Ran 1 test in 2.736s
2023-01-11T22:03:10.2633017Z 
2023-01-11T22:03:10.2633078Z OK
2023-01-11T22:03:10.2633083Z 
2023-01-11T22:03:10.2633167Z Generating XML reports...
2023-01-11T22:03:10.2633505Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214644.xml
2023-01-11T22:03:10.2633765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2633889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2634168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2634301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2634485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp91uppq62
2023-01-11T22:03:10.2634678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp91uppq62/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2634934Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2634940Z 
2023-01-11T22:03:10.2635015Z Running tests...
2023-01-11T22:03:10.2635211Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2635417Z   test_non_cont_tensors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21035
2023-01-11T22:03:10.2635572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21036
2023-01-11T22:03:10.2635725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21037
2023-01-11T22:03:10.2635878Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21038
2023-01-11T22:03:10.2636155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2636321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2636604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2636769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2636943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp160oqj5x
2023-01-11T22:03:10.2637138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp160oqj5x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2637414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2637539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2637822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2637958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2638146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvw6mon_q
2023-01-11T22:03:10.2638346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvw6mon_q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2638622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2638732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2639093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2639231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2639415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8yp7yhm
2023-01-11T22:03:10.2639609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8yp7yhm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2639889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2640014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2640294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2640430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2640606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpriydoulc
2023-01-11T22:03:10.2640802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpriydoulc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2640968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2641132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2641298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2641463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2641533Z ok (2.639s)
2023-01-11T22:03:10.2641540Z 
2023-01-11T22:03:10.2641739Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2641803Z Ran 1 test in 2.640s
2023-01-11T22:03:10.2641808Z 
2023-01-11T22:03:10.2641870Z OK
2023-01-11T22:03:10.2641876Z 
2023-01-11T22:03:10.2641961Z Generating XML reports...
2023-01-11T22:03:10.2642299Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214649.xml
2023-01-11T22:03:10.2642576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2642701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2642983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2643156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2643370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpud1c2yh6
2023-01-11T22:03:10.2643555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpud1c2yh6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2643811Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2643817Z 
2023-01-11T22:03:10.2643890Z Running tests...
2023-01-11T22:03:10.2644087Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2644357Z   test_non_garbage_collected_user_rref_due_to_local_circular_dependency (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21258
2023-01-11T22:03:10.2644516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21259
2023-01-11T22:03:10.2644672Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21260
2023-01-11T22:03:10.2644828Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21261
2023-01-11T22:03:10.2645091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2645214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2645495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2645629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2645814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5lba0qro
2023-01-11T22:03:10.2646009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5lba0qro/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2646281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2646406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2646688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2646812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2646997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcoh6hfl
2023-01-11T22:03:10.2647190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcoh6hfl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2647459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2647582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2647861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2647997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2648182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynlqeo5z
2023-01-11T22:03:10.2648364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynlqeo5z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2648632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2648754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2649033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2649166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2649351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hslhx4_
2023-01-11T22:03:10.2649579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hslhx4_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2649773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2649938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2650088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2650248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2650316Z ok (2.644s)
2023-01-11T22:03:10.2650321Z 
2023-01-11T22:03:10.2650520Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2650595Z Ran 1 test in 2.644s
2023-01-11T22:03:10.2650601Z 
2023-01-11T22:03:10.2650664Z OK
2023-01-11T22:03:10.2650669Z 
2023-01-11T22:03:10.2650752Z Generating XML reports...
2023-01-11T22:03:10.2651088Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214653.xml
2023-01-11T22:03:10.2651366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2651478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2651757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2651892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2652076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrf0zefj
2023-01-11T22:03:10.2652271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrf0zefj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2652525Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2652531Z 
2023-01-11T22:03:10.2652607Z Running tests...
2023-01-11T22:03:10.2652801Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2653003Z   test_nonzero (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21481
2023-01-11T22:03:10.2653158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21482
2023-01-11T22:03:10.2653311Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21483
2023-01-11T22:03:10.2653465Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21484
2023-01-11T22:03:10.2653741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2653864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2654146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2654283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2654470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptec8ctv6
2023-01-11T22:03:10.2654731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2654854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2655134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2655272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2655459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp278zhtnf
2023-01-11T22:03:10.2655655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptec8ctv6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2655847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp278zhtnf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2656155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2656312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2656594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2656729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2656913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6xnjokg
2023-01-11T22:03:10.2657111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6xnjokg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2657385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2657510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2657792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2657929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2658101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmq1dhi6
2023-01-11T22:03:10.2658295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmq1dhi6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2658459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2658624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2658786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2658946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2659016Z ok (2.640s)
2023-01-11T22:03:10.2659023Z 
2023-01-11T22:03:10.2659222Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2659285Z Ran 1 test in 2.640s
2023-01-11T22:03:10.2659291Z 
2023-01-11T22:03:10.2659354Z OK
2023-01-11T22:03:10.2659358Z 
2023-01-11T22:03:10.2659446Z Generating XML reports...
2023-01-11T22:03:10.2659787Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214658.xml
2023-01-11T22:03:10.2660064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2660189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2660472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2660613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2660799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcy59yin
2023-01-11T22:03:10.2660986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcy59yin/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2661245Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2661251Z 
2023-01-11T22:03:10.2661327Z Running tests...
2023-01-11T22:03:10.2661527Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2662027Z   test_owner_equality (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21704
2023-01-11T22:03:10.2662187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21705
2023-01-11T22:03:10.2662342Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21706
2023-01-11T22:03:10.2662503Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21707
2023-01-11T22:03:10.2662834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2662960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2663283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2663423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2663613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxqvlrti
2023-01-11T22:03:10.2663813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxqvlrti/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2664090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2664217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2664502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2664627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2664815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4c4rymlu
2023-01-11T22:03:10.2665012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4c4rymlu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2665292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2665417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2665702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2665837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2666024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_pr9pet
2023-01-11T22:03:10.2666210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_pr9pet/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2666484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2666609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2666888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2667022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2667206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_d2jppjd
2023-01-11T22:03:10.2667403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_d2jppjd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2667569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2667738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2667887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2668052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2668121Z ok (2.641s)
2023-01-11T22:03:10.2668127Z 
2023-01-11T22:03:10.2668327Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2668408Z Ran 1 test in 2.642s
2023-01-11T22:03:10.2668414Z 
2023-01-11T22:03:10.2668479Z OK
2023-01-11T22:03:10.2668484Z 
2023-01-11T22:03:10.2668570Z Generating XML reports...
2023-01-11T22:03:10.2668911Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214702.xml
2023-01-11T22:03:10.2669172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2669330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2669643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2669779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2669962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxqjs1lo
2023-01-11T22:03:10.2670155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxqjs1lo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2670416Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2670422Z 
2023-01-11T22:03:10.2670495Z Running tests...
2023-01-11T22:03:10.2670690Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2670900Z   test_owner_rref_backward (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21927
2023-01-11T22:03:10.2671059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21928
2023-01-11T22:03:10.2671214Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21929
2023-01-11T22:03:10.2671365Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21930
2023-01-11T22:03:10.2671645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2671769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2672050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2672187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2672360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwy0ud199
2023-01-11T22:03:10.2672555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwy0ud199/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2672828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2672954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2673235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2673370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2673554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn6bvidsc
2023-01-11T22:03:10.2673748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn6bvidsc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2674017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2674127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2674410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2674546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2674729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4so0tnt3
2023-01-11T22:03:10.2674921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4so0tnt3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2675190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2675312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2675590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2675723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2675928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9vmgz2_
2023-01-11T22:03:10.2676149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9vmgz2_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2676314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2676479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2676640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2676800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2676870Z ok (2.644s)
2023-01-11T22:03:10.2676875Z 
2023-01-11T22:03:10.2677072Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2677136Z Ran 1 test in 2.644s
2023-01-11T22:03:10.2677142Z 
2023-01-11T22:03:10.2677202Z OK
2023-01-11T22:03:10.2677209Z 
2023-01-11T22:03:10.2677293Z Generating XML reports...
2023-01-11T22:03:10.2677631Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214707.xml
2023-01-11T22:03:10.2677906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2678030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2678310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2678443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2678627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbusgg_u8
2023-01-11T22:03:10.2678809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbusgg_u8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2679132Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2679140Z 
2023-01-11T22:03:10.2679213Z Running tests...
2023-01-11T22:03:10.2679411Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2679633Z   test_pass_local_rrefs (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22170
2023-01-11T22:03:10.2679791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22171
2023-01-11T22:03:10.2679948Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22172
2023-01-11T22:03:10.2680101Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22173
2023-01-11T22:03:10.2680365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2680489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2680772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2680909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2681100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4gnaca7
2023-01-11T22:03:10.2681300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4gnaca7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2681575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2681698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2681979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2682102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2682289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpagbxhld_
2023-01-11T22:03:10.2682524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpagbxhld_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2682824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2682951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2683231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2683367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2683552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrfl0p5s
2023-01-11T22:03:10.2683740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrfl0p5s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2684013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2684137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2684421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2684554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2684739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphikpyvts
2023-01-11T22:03:10.2684934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphikpyvts/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2685098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2685262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2685412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2685574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2685644Z ok (2.740s)
2023-01-11T22:03:10.2685650Z 
2023-01-11T22:03:10.2685851Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2685926Z Ran 1 test in 2.740s
2023-01-11T22:03:10.2685931Z 
2023-01-11T22:03:10.2685992Z OK
2023-01-11T22:03:10.2685996Z 
2023-01-11T22:03:10.2686080Z Generating XML reports...
2023-01-11T22:03:10.2686419Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214712.xml
2023-01-11T22:03:10.2686680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2686804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2687083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2687221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2687406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpue_iqkvi
2023-01-11T22:03:10.2687602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpue_iqkvi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2687857Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2687863Z 
2023-01-11T22:03:10.2687936Z Running tests...
2023-01-11T22:03:10.2688135Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2688342Z   test_pg_init_no_rpc_init (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22393
2023-01-11T22:03:10.2688498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22394
2023-01-11T22:03:10.2688655Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22395
2023-01-11T22:03:10.2688879Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22396
2023-01-11T22:03:10.2689179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2689303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2689585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2689722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2689895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx48_ovll
2023-01-11T22:03:10.2690087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx48_ovll/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2690362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2690486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2690772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2690910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2691099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbzx06bm
2023-01-11T22:03:10.2691293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbzx06bm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2691564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2691674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2691953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2692089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2692275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1uxudcpx
2023-01-11T22:03:10.2692471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1uxudcpx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2692743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2692866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2693148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2693284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2693455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb9rqbu4w
2023-01-11T22:03:10.2693650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb9rqbu4w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2693817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2693982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2694145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2694311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2694486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.2694664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.2694827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.2695129Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2695431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2695665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.2695965Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2696263Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.2696331Z ok (2.450s)
2023-01-11T22:03:10.2696336Z 
2023-01-11T22:03:10.2696535Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2696610Z Ran 1 test in 2.450s
2023-01-11T22:03:10.2696616Z 
2023-01-11T22:03:10.2696663Z OK
2023-01-11T22:03:10.2696680Z 
2023-01-11T22:03:10.2696752Z Generating XML reports...
2023-01-11T22:03:10.2697091Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214716.xml
2023-01-11T22:03:10.2697368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2697494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2697776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2697913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2698101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt20zile1
2023-01-11T22:03:10.2698297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt20zile1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2698539Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2698545Z 
2023-01-11T22:03:10.2698618Z Running tests...
2023-01-11T22:03:10.2698818Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2699036Z   test_pickle_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22464
2023-01-11T22:03:10.2699195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22465
2023-01-11T22:03:10.2699349Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22466
2023-01-11T22:03:10.2699501Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22467
2023-01-11T22:03:10.2699777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2699888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2700173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2700308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2700497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31zp8osr
2023-01-11T22:03:10.2700696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31zp8osr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2700970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2701097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2701379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2701513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2701787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_g8tzmn
2023-01-11T22:03:10.2701986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_g8tzmn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2702314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2702491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2702776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2702912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2703103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpemihd9zq
2023-01-11T22:03:10.2703374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2703498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2703762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2703902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2704089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmr10dpj
2023-01-11T22:03:10.2704290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpemihd9zq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2704483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmr10dpj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2704650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2704813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2704976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2705125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2705193Z ok (2.641s)
2023-01-11T22:03:10.2705200Z 
2023-01-11T22:03:10.2705400Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2705476Z Ran 1 test in 2.641s
2023-01-11T22:03:10.2705484Z 
2023-01-11T22:03:10.2705548Z OK
2023-01-11T22:03:10.2705553Z 
2023-01-11T22:03:10.2705640Z Generating XML reports...
2023-01-11T22:03:10.2705983Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214721.xml
2023-01-11T22:03:10.2706261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2706384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2706653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2706788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2706973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpkwxv4k
2023-01-11T22:03:10.2707175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpkwxv4k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2707432Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2707437Z 
2023-01-11T22:03:10.2707511Z Running tests...
2023-01-11T22:03:10.2707707Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2707933Z   test_profiler_export_trace (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22687
2023-01-11T22:03:10.2708077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22688
2023-01-11T22:03:10.2708233Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22689
2023-01-11T22:03:10.2708386Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22690
2023-01-11T22:03:10.2708663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2708820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2709132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2709269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2709454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp926o06ex
2023-01-11T22:03:10.2709647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp926o06ex/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2709909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2710031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2710311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2710448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2710635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzkzeekj
2023-01-11T22:03:10.2710832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzkzeekj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2711103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2711226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2711494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2711626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2711808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu73c5ej2
2023-01-11T22:03:10.2712004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu73c5ej2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2712283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2712406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2712687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2712820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2713003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4kj49oo
2023-01-11T22:03:10.2713184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4kj49oo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2713348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2713511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2713677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2713843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2713911Z ok (2.648s)
2023-01-11T22:03:10.2713917Z 
2023-01-11T22:03:10.2714118Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2714194Z Ran 1 test in 2.648s
2023-01-11T22:03:10.2714200Z 
2023-01-11T22:03:10.2714248Z OK
2023-01-11T22:03:10.2714268Z 
2023-01-11T22:03:10.2714340Z Generating XML reports...
2023-01-11T22:03:10.2714680Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214725.xml
2023-01-11T22:03:10.2714954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2715080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2715399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2715561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2715749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppv3r092k
2023-01-11T22:03:10.2715941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppv3r092k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2716186Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2716191Z 
2023-01-11T22:03:10.2716267Z Running tests...
2023-01-11T22:03:10.2716467Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2716708Z   test_profiler_remote_events_profiled (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22910
2023-01-11T22:03:10.2716868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22911
2023-01-11T22:03:10.2717025Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22912
2023-01-11T22:03:10.2717181Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22913
2023-01-11T22:03:10.2717461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2717572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2717860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2717998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2718188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa58iu6nt
2023-01-11T22:03:10.2718466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2718594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2718943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2719083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2719270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4x5x9v1n
2023-01-11T22:03:10.2719454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa58iu6nt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2719649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4x5x9v1n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2719924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2720049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2720335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2720473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2720663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejn9zk1i
2023-01-11T22:03:10.2720861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejn9zk1i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2721135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2721247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2721527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2721663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2721853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5t2yghk
2023-01-11T22:03:10.2722103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5t2yghk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2722303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2722472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2722635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2722783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2722853Z ok (2.640s)
2023-01-11T22:03:10.2722860Z 
2023-01-11T22:03:10.2723062Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2723143Z Ran 1 test in 2.640s
2023-01-11T22:03:10.2723148Z 
2023-01-11T22:03:10.2723210Z OK
2023-01-11T22:03:10.2723216Z 
2023-01-11T22:03:10.2723304Z Generating XML reports...
2023-01-11T22:03:10.2723654Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214730.xml
2023-01-11T22:03:10.2723929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2724055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2724331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2724468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2724654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkn8uhnc1
2023-01-11T22:03:10.2724854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkn8uhnc1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2725131Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2725140Z 
2023-01-11T22:03:10.2725214Z Running tests...
2023-01-11T22:03:10.2725417Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2725675Z   test_profiler_remote_events_profiled_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23133
2023-01-11T22:03:10.2725822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23134
2023-01-11T22:03:10.2725975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23135
2023-01-11T22:03:10.2726127Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23136
2023-01-11T22:03:10.2726402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2726524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2726806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2726946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2727135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp126r7w4e
2023-01-11T22:03:10.2727328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp126r7w4e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2727588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2727710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2727993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2728130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2728315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9wkj8s9
2023-01-11T22:03:10.2728549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9wkj8s9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2728855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2728980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2729248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2729380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2729564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpfap56o
2023-01-11T22:03:10.2729834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2729957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2730239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2730374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2730559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95q2dh3z
2023-01-11T22:03:10.2730755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpfap56o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2730932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95q2dh3z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2731095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2731260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2731420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2731581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2731650Z ok (2.746s)
2023-01-11T22:03:10.2731656Z 
2023-01-11T22:03:10.2731857Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2731932Z Ran 1 test in 2.746s
2023-01-11T22:03:10.2731937Z 
2023-01-11T22:03:10.2731985Z OK
2023-01-11T22:03:10.2732001Z 
2023-01-11T22:03:10.2732072Z Generating XML reports...
2023-01-11T22:03:10.2732410Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214734.xml
2023-01-11T22:03:10.2732685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2732809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2733091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2733226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2733412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1j2u4v8
2023-01-11T22:03:10.2733608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1j2u4v8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2733850Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2733856Z 
2023-01-11T22:03:10.2733928Z Running tests...
2023-01-11T22:03:10.2734123Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2734355Z   test_profiler_rpc_key_names (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23356
2023-01-11T22:03:10.2734514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23357
2023-01-11T22:03:10.2734669Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23358
2023-01-11T22:03:10.2734859Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23359
2023-01-11T22:03:10.2735301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2735427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2735702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2735838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2736026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptmrabl9n
2023-01-11T22:03:10.2736225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptmrabl9n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2736504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2736630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2736920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2737062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2737238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpimuys8re
2023-01-11T22:03:10.2737511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2737636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2737917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2738052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2738236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12mdmv_l
2023-01-11T22:03:10.2738438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpimuys8re/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2738631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12mdmv_l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2738904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2739016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2739297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2739431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2739616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjngzeqt
2023-01-11T22:03:10.2739812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjngzeqt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2739978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2740142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2740304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2740452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2740523Z ok (2.640s)
2023-01-11T22:03:10.2740529Z 
2023-01-11T22:03:10.2740727Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2740801Z Ran 1 test in 2.640s
2023-01-11T22:03:10.2740807Z 
2023-01-11T22:03:10.2740868Z OK
2023-01-11T22:03:10.2740873Z 
2023-01-11T22:03:10.2740957Z Generating XML reports...
2023-01-11T22:03:10.2741297Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214739.xml
2023-01-11T22:03:10.2741572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2741838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2742162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2742301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2742490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kxp59uq
2023-01-11T22:03:10.2742689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kxp59uq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2742946Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2742952Z 
2023-01-11T22:03:10.2743027Z Running tests...
2023-01-11T22:03:10.2743226Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2743454Z   test_profiler_rpc_memory (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23585
2023-01-11T22:03:10.2743600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23586
2023-01-11T22:03:10.2743755Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23587
2023-01-11T22:03:10.2743911Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23588
2023-01-11T22:03:10.2744187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2744311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2744597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2744733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2744919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8b4tx7q
2023-01-11T22:03:10.2745115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8b4tx7q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2745379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2745503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2745785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2745921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2746106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb49vdox5
2023-01-11T22:03:10.2746301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb49vdox5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2746570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2746695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2746965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2747099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2747285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnrgvbe25
2023-01-11T22:03:10.2747480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnrgvbe25/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2747752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2747876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2748157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2748334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2748555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93i5_sl6
2023-01-11T22:03:10.2748736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93i5_sl6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2748901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2749066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2749229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2749390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2749459Z ok (2.648s)
2023-01-11T22:03:10.2749465Z 
2023-01-11T22:03:10.2749663Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2749742Z Ran 1 test in 2.649s
2023-01-11T22:03:10.2749748Z 
2023-01-11T22:03:10.2749796Z OK
2023-01-11T22:03:10.2749816Z 
2023-01-11T22:03:10.2749886Z Generating XML reports...
2023-01-11T22:03:10.2750229Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214743.xml
2023-01-11T22:03:10.2750503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2750626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2750904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2751038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2751225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8nmio33f
2023-01-11T22:03:10.2751421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8nmio33f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2751666Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2751671Z 
2023-01-11T22:03:10.2751745Z Running tests...
2023-01-11T22:03:10.2751940Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2752174Z   test_profiler_rpc_record_shapes (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23808
2023-01-11T22:03:10.2752331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23809
2023-01-11T22:03:10.2752489Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23810
2023-01-11T22:03:10.2752642Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23811
2023-01-11T22:03:10.2752919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2753045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2753316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2753451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2753637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvdb9l4bv
2023-01-11T22:03:10.2753833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvdb9l4bv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2754106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2754231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2754509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2754646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2754853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd0xkfwcw
2023-01-11T22:03:10.2755079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd0xkfwcw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2755355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2755481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2755761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2755896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2756084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pyux9m_
2023-01-11T22:03:10.2756355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2756480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2756750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2756884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2757068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjaq8itqy
2023-01-11T22:03:10.2757264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pyux9m_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2757454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjaq8itqy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2757618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2757782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2757947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2758095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2758164Z ok (2.658s)
2023-01-11T22:03:10.2758170Z 
2023-01-11T22:03:10.2758367Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2758445Z Ran 1 test in 2.659s
2023-01-11T22:03:10.2758450Z 
2023-01-11T22:03:10.2758512Z OK
2023-01-11T22:03:10.2758517Z 
2023-01-11T22:03:10.2758601Z Generating XML reports...
2023-01-11T22:03:10.2759008Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214748.xml
2023-01-11T22:03:10.2759285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2759409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2759677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2759816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2760005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnrczkptr
2023-01-11T22:03:10.2760203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnrczkptr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2760461Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2760466Z 
2023-01-11T22:03:10.2760539Z Running tests...
2023-01-11T22:03:10.2760738Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2760975Z   test_profiler_with_async_rpc_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24031
2023-01-11T22:03:10.2761120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24032
2023-01-11T22:03:10.2761319Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24033
2023-01-11T22:03:10.2761500Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24034
2023-01-11T22:03:10.2761779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2761904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2762190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2762324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2762508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__eko6aa
2023-01-11T22:03:10.2762701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__eko6aa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2762962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2763109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2763465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2763600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2763786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngqqtf6r
2023-01-11T22:03:10.2763982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngqqtf6r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2764259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2764384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2764650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2764826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2765002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt705qedm
2023-01-11T22:03:10.2765198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt705qedm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2765474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2765598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2765880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2766013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2766195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhl3z808
2023-01-11T22:03:10.2766390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhl3z808/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2766543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2766706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2766867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2767030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2767098Z ok (2.741s)
2023-01-11T22:03:10.2767104Z 
2023-01-11T22:03:10.2767303Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2767379Z Ran 1 test in 2.741s
2023-01-11T22:03:10.2767384Z 
2023-01-11T22:03:10.2767444Z OK
2023-01-11T22:03:10.2767448Z 
2023-01-11T22:03:10.2767535Z Generating XML reports...
2023-01-11T22:03:10.2767863Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214753.xml
2023-01-11T22:03:10.2768213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2768340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2768623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2768758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2768944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7q9zcysl
2023-01-11T22:03:10.2769138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7q9zcysl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2769395Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2769401Z 
2023-01-11T22:03:10.2769464Z Running tests...
2023-01-11T22:03:10.2769663Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2769919Z   test_profiler_with_async_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24254
2023-01-11T22:03:10.2770079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24255
2023-01-11T22:03:10.2770235Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24256
2023-01-11T22:03:10.2770390Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24257
2023-01-11T22:03:10.2770669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2770797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2771085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2771212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2771403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8adrnbxw
2023-01-11T22:03:10.2771604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8adrnbxw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2771882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2772009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2772293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2772430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2772616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd24mcz_d
2023-01-11T22:03:10.2772801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd24mcz_d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2773080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2773205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2773491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2773628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2773816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpirkxjw2k
2023-01-11T22:03:10.2774014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpirkxjw2k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2774288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2774412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2774711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2774879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2775064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8p80_c6
2023-01-11T22:03:10.2775260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8p80_c6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2775427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2775592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2775756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2775920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2775978Z ok (2.538s)
2023-01-11T22:03:10.2775997Z 
2023-01-11T22:03:10.2776184Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2776261Z Ran 1 test in 2.538s
2023-01-11T22:03:10.2776269Z 
2023-01-11T22:03:10.2776333Z OK
2023-01-11T22:03:10.2776339Z 
2023-01-11T22:03:10.2776424Z Generating XML reports...
2023-01-11T22:03:10.2776767Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214757.xml
2023-01-11T22:03:10.2777045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2777170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2777451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2777575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2777763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpub0iepji
2023-01-11T22:03:10.2777963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpub0iepji/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2778220Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2778226Z 
2023-01-11T22:03:10.2778301Z Running tests...
2023-01-11T22:03:10.2778497Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2778728Z   test_profiler_with_async_rpc_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24477
2023-01-11T22:03:10.2778887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24478
2023-01-11T22:03:10.2779042Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24479
2023-01-11T22:03:10.2779183Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24480
2023-01-11T22:03:10.2779461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2779587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2779871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2780009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2780195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbmll67_
2023-01-11T22:03:10.2780390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbmll67_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2780662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2780774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2781087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2781223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2781444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcrj_u18
2023-01-11T22:03:10.2781639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcrj_u18/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2782024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2782155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2782435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2782571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2782744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoj4b1_do
2023-01-11T22:03:10.2783020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2783145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2783429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2783564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2783753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifgh3lq7
2023-01-11T22:03:10.2783949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoj4b1_do/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2784144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifgh3lq7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2784295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2784464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2784630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2784792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2785050Z STAGE:2023-01-11 21:48:06 24478:24478 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:03:10.2785283Z [W utils.cpp:175] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator())
2023-01-11T22:03:10.2785540Z STAGE:2023-01-11 21:48:07 24478:24478 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:03:10.2785809Z STAGE:2023-01-11 21:48:07 24478:24478 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:03:10.2785876Z ok (5.646s)
2023-01-11T22:03:10.2785882Z 
2023-01-11T22:03:10.2786072Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2786147Z Ran 1 test in 5.646s
2023-01-11T22:03:10.2786153Z 
2023-01-11T22:03:10.2786219Z OK
2023-01-11T22:03:10.2786226Z 
2023-01-11T22:03:10.2786312Z Generating XML reports...
2023-01-11T22:03:10.2786656Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214802.xml
2023-01-11T22:03:10.2786936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2787062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2787341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2787475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2787648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqsan_0i
2023-01-11T22:03:10.2787917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqsan_0i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2788246Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2788253Z 
2023-01-11T22:03:10.2788330Z Running tests...
2023-01-11T22:03:10.2788534Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2788792Z   test_profiler_with_async_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24701
2023-01-11T22:03:10.2788952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24702
2023-01-11T22:03:10.2789113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24703
2023-01-11T22:03:10.2789258Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24704
2023-01-11T22:03:10.2789544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2789673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2789976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2790118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2790312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeb4dj_ip
2023-01-11T22:03:10.2790515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeb4dj_ip/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2790796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2790921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2791194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2791389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2791660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb67zsuki
2023-01-11T22:03:10.2791999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2792124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2792406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2792548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2792737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpx3ah27
2023-01-11T22:03:10.2792923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb67zsuki/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2793125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpx3ah27/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2793408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2793540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2793832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2793978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2794168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_8z7_ae
2023-01-11T22:03:10.2794365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_8z7_ae/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2794537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2794750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2794917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2795130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2795389Z STAGE:2023-01-11 21:48:14 24702:24702 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:03:10.2795615Z [W utils.cpp:175] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator())
2023-01-11T22:03:10.2795872Z STAGE:2023-01-11 21:48:15 24702:24702 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:03:10.2796133Z STAGE:2023-01-11 21:48:15 24702:24702 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:03:10.2796201Z ok (5.646s)
2023-01-11T22:03:10.2796208Z 
2023-01-11T22:03:10.2796408Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2796475Z Ran 1 test in 5.646s
2023-01-11T22:03:10.2796480Z 
2023-01-11T22:03:10.2796541Z OK
2023-01-11T22:03:10.2796548Z 
2023-01-11T22:03:10.2796635Z Generating XML reports...
2023-01-11T22:03:10.2796982Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214809.xml
2023-01-11T22:03:10.2797261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2797389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2797682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2805819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2806117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoy9cprg1
2023-01-11T22:03:10.2806319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoy9cprg1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2806628Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2806636Z 
2023-01-11T22:03:10.2806712Z Running tests...
2023-01-11T22:03:10.2806917Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2807163Z   test_profiler_with_autograd_context (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24925
2023-01-11T22:03:10.2807326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24926
2023-01-11T22:03:10.2807484Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24927
2023-01-11T22:03:10.2807641Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24928
2023-01-11T22:03:10.2807928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2808045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2808335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2808473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2808661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehzn055w
2023-01-11T22:03:10.2808858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehzn055w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2809137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2809262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2809545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2809817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2810045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcv_0ts9b
2023-01-11T22:03:10.2810242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcv_0ts9b/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2810518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2810644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2810927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2811063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2811251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph745rvee
2023-01-11T22:03:10.2811446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph745rvee/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2811709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2811833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2812115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2812252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2812437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lwgob7n
2023-01-11T22:03:10.2812633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lwgob7n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2812798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2812962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2813115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2813282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2813352Z ok (2.660s)
2023-01-11T22:03:10.2813358Z 
2023-01-11T22:03:10.2813560Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2813638Z Ran 1 test in 2.660s
2023-01-11T22:03:10.2813644Z 
2023-01-11T22:03:10.2813706Z OK
2023-01-11T22:03:10.2813711Z 
2023-01-11T22:03:10.2813797Z Generating XML reports...
2023-01-11T22:03:10.2814144Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214817.xml
2023-01-11T22:03:10.2814417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2814529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2814812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2814947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2815134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzpo3zo9
2023-01-11T22:03:10.2815329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzpo3zo9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2815579Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2815585Z 
2023-01-11T22:03:10.2815660Z Running tests...
2023-01-11T22:03:10.2815859Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2816100Z   test_profiler_with_autograd_context_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25148
2023-01-11T22:03:10.2816291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25149
2023-01-11T22:03:10.2816447Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25150
2023-01-11T22:03:10.2816630Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25151
2023-01-11T22:03:10.2816915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2817040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2817331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2817467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2817654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplsj7auc5
2023-01-11T22:03:10.2817839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplsj7auc5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2818120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2818247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2818531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2818667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2818853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9vv7u196
2023-01-11T22:03:10.2819047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9vv7u196/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2819320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2819444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2819715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2819854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2820040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxk10ruj
2023-01-11T22:03:10.2820237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxk10ruj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2820510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2820635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2820916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2821054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2821227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprh2nhpyc
2023-01-11T22:03:10.2821427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprh2nhpyc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2821592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2821920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2822085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2822248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2822319Z ok (2.748s)
2023-01-11T22:03:10.2822325Z 
2023-01-11T22:03:10.2822528Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2822605Z Ran 1 test in 2.749s
2023-01-11T22:03:10.2822610Z 
2023-01-11T22:03:10.2822658Z OK
2023-01-11T22:03:10.2822663Z 
2023-01-11T22:03:10.2822822Z Generating XML reports...
2023-01-11T22:03:10.2823167Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214822.xml
2023-01-11T22:03:10.2823479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2823606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2823889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2824029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2824218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuybwylpp
2023-01-11T22:03:10.2824406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuybwylpp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2824665Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2824674Z 
2023-01-11T22:03:10.2824750Z Running tests...
2023-01-11T22:03:10.2824950Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2825186Z   test_profiler_with_remote_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25371
2023-01-11T22:03:10.2825345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25372
2023-01-11T22:03:10.2825502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25373
2023-01-11T22:03:10.2825659Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25374
2023-01-11T22:03:10.2825937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2826050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2826334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2826473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2826664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3g92dej
2023-01-11T22:03:10.2826864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3g92dej/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2827144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2827269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2827551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2827675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2827860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xhpkz7p
2023-01-11T22:03:10.2828060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xhpkz7p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2828335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2828459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2828740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2828878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2829064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3968cv6e
2023-01-11T22:03:10.2829257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3968cv6e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2829515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2829672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2829981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2830119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2830308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppydgrhzj
2023-01-11T22:03:10.2830506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppydgrhzj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2830672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2830837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2831000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2831149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2831219Z ok (2.644s)
2023-01-11T22:03:10.2831225Z 
2023-01-11T22:03:10.2831426Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2831504Z Ran 1 test in 2.644s
2023-01-11T22:03:10.2831510Z 
2023-01-11T22:03:10.2831571Z OK
2023-01-11T22:03:10.2831577Z 
2023-01-11T22:03:10.2831661Z Generating XML reports...
2023-01-11T22:03:10.2832004Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214826.xml
2023-01-11T22:03:10.2832280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2832391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2832672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2832807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2832995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmqbj_j9
2023-01-11T22:03:10.2833192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmqbj_j9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2833449Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2833455Z 
2023-01-11T22:03:10.2833530Z Running tests...
2023-01-11T22:03:10.2833729Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2833977Z   test_profiler_with_remote_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25594
2023-01-11T22:03:10.2834122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25595
2023-01-11T22:03:10.2834280Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25596
2023-01-11T22:03:10.2834437Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25597
2023-01-11T22:03:10.2834713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2834840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2835123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2835262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2835447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8_83i3_
2023-01-11T22:03:10.2835627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8_83i3_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2835900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2836024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2836361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2836527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2836715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdc8kcy_l
2023-01-11T22:03:10.2836916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdc8kcy_l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2837192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2837316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2837582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2837717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2837904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplaj2uh0v
2023-01-11T22:03:10.2838104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplaj2uh0v/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2838377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2838504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2838783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2839012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2839186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpus4_br9v
2023-01-11T22:03:10.2839383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpus4_br9v/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2839553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2839718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2839886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2840050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2840119Z ok (2.740s)
2023-01-11T22:03:10.2840124Z 
2023-01-11T22:03:10.2840329Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2840405Z Ran 1 test in 2.741s
2023-01-11T22:03:10.2840410Z 
2023-01-11T22:03:10.2840458Z OK
2023-01-11T22:03:10.2840463Z 
2023-01-11T22:03:10.2840548Z Generating XML reports...
2023-01-11T22:03:10.2840892Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214831.xml
2023-01-11T22:03:10.2841167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2841295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2841579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2841717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2841907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1l4bc4x
2023-01-11T22:03:10.2842092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1l4bc4x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2842350Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2842356Z 
2023-01-11T22:03:10.2842430Z Running tests...
2023-01-11T22:03:10.2842629Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2842903Z   test_profiler_with_remote_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25817
2023-01-11T22:03:10.2843092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25818
2023-01-11T22:03:10.2843251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25819
2023-01-11T22:03:10.2843406Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25820
2023-01-11T22:03:10.2843684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2843795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2844079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2844215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2844404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfgvi8zb
2023-01-11T22:03:10.2844602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfgvi8zb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2844878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2845004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2845290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2845427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2845599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgauvm2n
2023-01-11T22:03:10.2845796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgauvm2n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2846071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2846199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2846479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2846614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2846799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpllfuk4rx
2023-01-11T22:03:10.2846993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpllfuk4rx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2847252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2847376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2847656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2847794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2847981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpum242057
2023-01-11T22:03:10.2848172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpum242057/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2848336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2848501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2848663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2848814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2848882Z ok (5.753s)
2023-01-11T22:03:10.2848887Z 
2023-01-11T22:03:10.2849084Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2849192Z Ran 1 test in 5.754s
2023-01-11T22:03:10.2849198Z 
2023-01-11T22:03:10.2849260Z OK
2023-01-11T22:03:10.2849265Z 
2023-01-11T22:03:10.2849350Z Generating XML reports...
2023-01-11T22:03:10.2849734Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214835.xml
2023-01-11T22:03:10.2850010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2850121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2850400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2850537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2850722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxiutu_8j
2023-01-11T22:03:10.2850917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxiutu_8j/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2851176Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2851182Z 
2023-01-11T22:03:10.2851258Z Running tests...
2023-01-11T22:03:10.2851460Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2851708Z   test_profiler_with_remote_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26040
2023-01-11T22:03:10.2851852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26041
2023-01-11T22:03:10.2852008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26042
2023-01-11T22:03:10.2852160Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26043
2023-01-11T22:03:10.2852435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2852562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2852847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2852984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2853171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4ry5ozt
2023-01-11T22:03:10.2853353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4ry5ozt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2853629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2853753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2854036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2854172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2854360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjaidftk
2023-01-11T22:03:10.2854560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjaidftk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2854840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2854964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2855233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2855369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2855554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpga_810yv
2023-01-11T22:03:10.2855748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpga_810yv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2856058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2856216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2856499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2856636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2856807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl01jn4oj
2023-01-11T22:03:10.2857001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl01jn4oj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2857165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2857331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2857494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2857653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2857727Z ok (5.651s)
2023-01-11T22:03:10.2857733Z 
2023-01-11T22:03:10.2857930Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2858007Z Ran 1 test in 5.651s
2023-01-11T22:03:10.2858012Z 
2023-01-11T22:03:10.2858060Z OK
2023-01-11T22:03:10.2858066Z 
2023-01-11T22:03:10.2858150Z Generating XML reports...
2023-01-11T22:03:10.2858492Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214843.xml
2023-01-11T22:03:10.2858765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2858891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2859174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2859316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2859504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzuc9ctpy
2023-01-11T22:03:10.2859687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzuc9ctpy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2859943Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2859948Z 
2023-01-11T22:03:10.2860022Z Running tests...
2023-01-11T22:03:10.2860220Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2860455Z   test_profiler_with_script_async_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26263
2023-01-11T22:03:10.2860615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26264
2023-01-11T22:03:10.2860774Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26265
2023-01-11T22:03:10.2860931Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26266
2023-01-11T22:03:10.2861209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2861320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2861602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2861850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2862037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpla_md4h6
2023-01-11T22:03:10.2862232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpla_md4h6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2862560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2862686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2863006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2863145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2863317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvi44evjw
2023-01-11T22:03:10.2863512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvi44evjw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2863786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2863911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2864192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2864331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2864517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_9bmb4o
2023-01-11T22:03:10.2864710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_9bmb4o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2864969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2865095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2865377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2865512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2865696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwaj14t9
2023-01-11T22:03:10.2865894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwaj14t9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2866062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2866230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2866392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2866539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2866608Z ok (2.651s)
2023-01-11T22:03:10.2866614Z 
2023-01-11T22:03:10.2866812Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2866889Z Ran 1 test in 2.651s
2023-01-11T22:03:10.2866894Z 
2023-01-11T22:03:10.2866956Z OK
2023-01-11T22:03:10.2866961Z 
2023-01-11T22:03:10.2867046Z Generating XML reports...
2023-01-11T22:03:10.2867384Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214851.xml
2023-01-11T22:03:10.2867663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2867775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2868056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2868190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2868376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk646jcfj
2023-01-11T22:03:10.2868573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk646jcfj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2868828Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2868834Z 
2023-01-11T22:03:10.2868949Z Running tests...
2023-01-11T22:03:10.2869146Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2869429Z   test_profiler_with_script_async_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26486
2023-01-11T22:03:10.2869575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26487
2023-01-11T22:03:10.2869732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26488
2023-01-11T22:03:10.2869887Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26489
2023-01-11T22:03:10.2870168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2870293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2870579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2870719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2870907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg6ucl76x
2023-01-11T22:03:10.2871092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg6ucl76x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2871367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2871492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2871774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2871913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2872101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnd69s6gv
2023-01-11T22:03:10.2872297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnd69s6gv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2872573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2872698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2872965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2873102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2873283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7b3thy5
2023-01-11T22:03:10.2873478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7b3thy5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2873751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2873880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2874166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2874307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2874494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgivul0v
2023-01-11T22:03:10.2874676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgivul0v/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2874841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2875005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2875168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2875330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2875433Z ok (2.647s)
2023-01-11T22:03:10.2875439Z 
2023-01-11T22:03:10.2875636Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2875712Z Ran 1 test in 2.647s
2023-01-11T22:03:10.2875746Z 
2023-01-11T22:03:10.2875795Z OK
2023-01-11T22:03:10.2875799Z 
2023-01-11T22:03:10.2875884Z Generating XML reports...
2023-01-11T22:03:10.2876228Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214855.xml
2023-01-11T22:03:10.2876503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2876627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2876908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2877042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2877231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6m3ic0o
2023-01-11T22:03:10.2877427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6m3ic0o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2877671Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2877676Z 
2023-01-11T22:03:10.2877751Z Running tests...
2023-01-11T22:03:10.2877947Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2878184Z   test_profiler_with_script_remote_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26709
2023-01-11T22:03:10.2878341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26710
2023-01-11T22:03:10.2878498Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26711
2023-01-11T22:03:10.2878652Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26712
2023-01-11T22:03:10.2879010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2879125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2879419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2879555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2879742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj0emz9a6
2023-01-11T22:03:10.2879940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj0emz9a6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2880217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2880344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2880628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2880766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2880938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4u3cuo6
2023-01-11T22:03:10.2881134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4u3cuo6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2881408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2881532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2881811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2881946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2882133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp41poduu9
2023-01-11T22:03:10.2882367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp41poduu9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2882658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2882785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2883066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2883202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2883388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0uu25m1
2023-01-11T22:03:10.2883581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0uu25m1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2883748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2883915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2884080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2884229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2884297Z ok (2.638s)
2023-01-11T22:03:10.2884305Z 
2023-01-11T22:03:10.2884503Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2884579Z Ran 1 test in 2.638s
2023-01-11T22:03:10.2884585Z 
2023-01-11T22:03:10.2884647Z OK
2023-01-11T22:03:10.2884653Z 
2023-01-11T22:03:10.2884737Z Generating XML reports...
2023-01-11T22:03:10.2885077Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214900.xml
2023-01-11T22:03:10.2885350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2885464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2885747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2885882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2886072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphx3rryrt
2023-01-11T22:03:10.2886270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphx3rryrt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2886527Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2886532Z 
2023-01-11T22:03:10.2886606Z Running tests...
2023-01-11T22:03:10.2886804Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2887057Z   test_profiler_with_script_remote_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26932
2023-01-11T22:03:10.2887205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26933
2023-01-11T22:03:10.2887363Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26934
2023-01-11T22:03:10.2887519Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26935
2023-01-11T22:03:10.2887793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2887917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2888203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2888340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2888528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy00euzsh
2023-01-11T22:03:10.2888759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy00euzsh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2889046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2889173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2889454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2889589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2889775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2njycsvf
2023-01-11T22:03:10.2889971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2njycsvf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2890243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2890371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2890638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2890776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2890960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbzk3i_e4
2023-01-11T22:03:10.2891154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbzk3i_e4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2891426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2891552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2891833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2891971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2892156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwaf65gd
2023-01-11T22:03:10.2892340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwaf65gd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2892508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2892675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2892838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2893000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2893069Z ok (2.645s)
2023-01-11T22:03:10.2893074Z 
2023-01-11T22:03:10.2893272Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2893349Z Ran 1 test in 2.646s
2023-01-11T22:03:10.2893357Z 
2023-01-11T22:03:10.2893405Z OK
2023-01-11T22:03:10.2893409Z 
2023-01-11T22:03:10.2893494Z Generating XML reports...
2023-01-11T22:03:10.2893834Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214904.xml
2023-01-11T22:03:10.2894109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2894233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2894512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2894648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2894834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmizekhg
2023-01-11T22:03:10.2895031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmizekhg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2895308Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2895314Z 
2023-01-11T22:03:10.2895389Z Running tests...
2023-01-11T22:03:10.2895630Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2895867Z   test_profiler_with_script_sync_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27155
2023-01-11T22:03:10.2896026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27156
2023-01-11T22:03:10.2896182Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27157
2023-01-11T22:03:10.2896337Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27158
2023-01-11T22:03:10.2896613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2896723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2897007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2897146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2897332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2rzxjhh
2023-01-11T22:03:10.2897527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2rzxjhh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2897805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2897930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2898211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2898349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2898525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd72f3czf
2023-01-11T22:03:10.2898722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd72f3czf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2898993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2899118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2899402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2899539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2899724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqldiizn
2023-01-11T22:03:10.2899922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqldiizn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2900182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2900306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2900586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2900723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2900909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvthjh1oi
2023-01-11T22:03:10.2901105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvthjh1oi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2901270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2901435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2901598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2901947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2902018Z ok (2.638s)
2023-01-11T22:03:10.2902074Z 
2023-01-11T22:03:10.2902285Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2902362Z Ran 1 test in 2.639s
2023-01-11T22:03:10.2902368Z 
2023-01-11T22:03:10.2902431Z OK
2023-01-11T22:03:10.2902436Z 
2023-01-11T22:03:10.2902522Z Generating XML reports...
2023-01-11T22:03:10.2902862Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214909.xml
2023-01-11T22:03:10.2903139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2903250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2903533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2903675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2903862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_78a4kw3
2023-01-11T22:03:10.2904060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_78a4kw3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2904316Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2904322Z 
2023-01-11T22:03:10.2904396Z Running tests...
2023-01-11T22:03:10.2904596Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2904850Z   test_profiler_with_script_sync_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27378
2023-01-11T22:03:10.2904995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27379
2023-01-11T22:03:10.2905156Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27380
2023-01-11T22:03:10.2905310Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27381
2023-01-11T22:03:10.2905589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2905713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2905996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2906134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2906321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0sexz_nb
2023-01-11T22:03:10.2906515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0sexz_nb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2906775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2906901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2907183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2907318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2907505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvqsd35e
2023-01-11T22:03:10.2907699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvqsd35e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2907974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2908100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2908364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2908540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2908752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_jzvfoz
2023-01-11T22:03:10.2908949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_jzvfoz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2909222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2909346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2909626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2909763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2909949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6cbewq3
2023-01-11T22:03:10.2910132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6cbewq3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2910296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2910465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2910630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2910792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2910862Z ok (2.645s)
2023-01-11T22:03:10.2910868Z 
2023-01-11T22:03:10.2911066Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2911143Z Ran 1 test in 2.645s
2023-01-11T22:03:10.2911148Z 
2023-01-11T22:03:10.2911197Z OK
2023-01-11T22:03:10.2911215Z 
2023-01-11T22:03:10.2911287Z Generating XML reports...
2023-01-11T22:03:10.2911627Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214914.xml
2023-01-11T22:03:10.2911909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2912034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2912315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2912450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2912637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpru3svv0a
2023-01-11T22:03:10.2912830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpru3svv0a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2913073Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2913078Z 
2023-01-11T22:03:10.2913153Z Running tests...
2023-01-11T22:03:10.2913357Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2913594Z   test_profiler_with_sync_rpc_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27601
2023-01-11T22:03:10.2913753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27602
2023-01-11T22:03:10.2913910Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27603
2023-01-11T22:03:10.2914065Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27604
2023-01-11T22:03:10.2914341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2914452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2914735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2914904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2915089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_nh3j83k
2023-01-11T22:03:10.2915313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_nh3j83k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2915590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2915715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2915997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2916134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2916306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn56bw5j1
2023-01-11T22:03:10.2916502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn56bw5j1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2916777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2916903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2917185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2917320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2917505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp691l8pbf
2023-01-11T22:03:10.2917784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2917910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2918176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2918315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2918500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5vtjcn8y
2023-01-11T22:03:10.2918697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp691l8pbf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2918963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5vtjcn8y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2919133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2919300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2919462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2919613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2919682Z ok (2.650s)
2023-01-11T22:03:10.2919690Z 
2023-01-11T22:03:10.2919892Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2919970Z Ran 1 test in 2.651s
2023-01-11T22:03:10.2919975Z 
2023-01-11T22:03:10.2920040Z OK
2023-01-11T22:03:10.2920045Z 
2023-01-11T22:03:10.2920129Z Generating XML reports...
2023-01-11T22:03:10.2920475Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214918.xml
2023-01-11T22:03:10.2920750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2920875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2921144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2921281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2921466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9id1hjck
2023-01-11T22:03:10.2921707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9id1hjck/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2921995Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2922001Z 
2023-01-11T22:03:10.2922077Z Running tests...
2023-01-11T22:03:10.2922277Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2922527Z   test_profiler_with_sync_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27824
2023-01-11T22:03:10.2922672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27825
2023-01-11T22:03:10.2922829Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27826
2023-01-11T22:03:10.2922985Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27827
2023-01-11T22:03:10.2923265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2923392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2923678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2923816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2924002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcqi_ggnn
2023-01-11T22:03:10.2924198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcqi_ggnn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2924460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2924586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2924870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2925008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2925194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvk_541i0
2023-01-11T22:03:10.2925387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvk_541i0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2925663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2925787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2926056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2926191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2926374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3bufnhwy
2023-01-11T22:03:10.2926571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3bufnhwy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2926844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2926968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2927247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2927382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2927568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsg3s9yql
2023-01-11T22:03:10.2927749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsg3s9yql/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2927914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2928111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2928300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2928466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2928534Z ok (2.637s)
2023-01-11T22:03:10.2928540Z 
2023-01-11T22:03:10.2928740Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2928816Z Ran 1 test in 2.637s
2023-01-11T22:03:10.2928821Z 
2023-01-11T22:03:10.2928869Z OK
2023-01-11T22:03:10.2928886Z 
2023-01-11T22:03:10.2928958Z Generating XML reports...
2023-01-11T22:03:10.2929297Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214923.xml
2023-01-11T22:03:10.2929571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2929699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2929980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2930116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2930302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrx68nli
2023-01-11T22:03:10.2930498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrx68nli/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2930741Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2930747Z 
2023-01-11T22:03:10.2930821Z Running tests...
2023-01-11T22:03:10.2931019Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2931250Z   test_profiler_with_sync_rpc_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28047
2023-01-11T22:03:10.2931413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28048
2023-01-11T22:03:10.2931571Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28049
2023-01-11T22:03:10.2931726Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28050
2023-01-11T22:03:10.2932002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2932126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2932395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2932530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2932715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxu956tbm
2023-01-11T22:03:10.2932911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxu956tbm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2933188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2933313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2933593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2933727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2933895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2y__sf_d
2023-01-11T22:03:10.2934087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2y__sf_d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2934360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2934484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2934804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2934966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2935152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5t0zhz4
2023-01-11T22:03:10.2935347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5t0zhz4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2935624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2935734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2936013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2936147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2936332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6i9zkdex
2023-01-11T22:03:10.2936528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6i9zkdex/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2936693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2936858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2937022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2937172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2937240Z ok (4.649s)
2023-01-11T22:03:10.2937246Z 
2023-01-11T22:03:10.2937445Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2937522Z Ran 1 test in 4.650s
2023-01-11T22:03:10.2937526Z 
2023-01-11T22:03:10.2937591Z OK
2023-01-11T22:03:10.2937596Z 
2023-01-11T22:03:10.2937682Z Generating XML reports...
2023-01-11T22:03:10.2938025Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214927.xml
2023-01-11T22:03:10.2938302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2938426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2938692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2938827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2939010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_h2vh9z
2023-01-11T22:03:10.2939204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_h2vh9z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2939460Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2939467Z 
2023-01-11T22:03:10.2939544Z Running tests...
2023-01-11T22:03:10.2939743Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2939992Z   test_profiler_with_sync_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28270
2023-01-11T22:03:10.2940137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28271
2023-01-11T22:03:10.2940294Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28272
2023-01-11T22:03:10.2940449Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28273
2023-01-11T22:03:10.2940722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2940847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2941171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2941348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2941536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpts9v8hn4
2023-01-11T22:03:10.2941842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpts9v8hn4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2942104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2942228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2942511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2942645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2942836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_bfw9kv
2023-01-11T22:03:10.2943031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_bfw9kv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2943307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2943432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2943700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2943838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2944023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8xq3a2a
2023-01-11T22:03:10.2944218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8xq3a2a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2944492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2944618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2944904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2945043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2945228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvblu33kw
2023-01-11T22:03:10.2945408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvblu33kw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2945574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2945738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2945903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2946069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2946137Z ok (4.645s)
2023-01-11T22:03:10.2946143Z 
2023-01-11T22:03:10.2946346Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2946423Z Ran 1 test in 4.645s
2023-01-11T22:03:10.2946428Z 
2023-01-11T22:03:10.2946478Z OK
2023-01-11T22:03:10.2946497Z 
2023-01-11T22:03:10.2946568Z Generating XML reports...
2023-01-11T22:03:10.2946910Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214934.xml
2023-01-11T22:03:10.2947184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2947308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2947590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2947780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2947998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ljlf645
2023-01-11T22:03:10.2948194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ljlf645/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2948437Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2948458Z 
2023-01-11T22:03:10.2948519Z Running tests...
2023-01-11T22:03:10.2948716Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2948932Z   test_py_built_in (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28493
2023-01-11T22:03:10.2949092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28494
2023-01-11T22:03:10.2949250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28495
2023-01-11T22:03:10.2949405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28496
2023-01-11T22:03:10.2949682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2949807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2950076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2950211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2950395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp16sro2zi
2023-01-11T22:03:10.2950586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp16sro2zi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2950860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2950986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2951268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2951403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2951572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpje_gu0lw
2023-01-11T22:03:10.2951765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpje_gu0lw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2952034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2952157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2952437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2952572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2952758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplyrcy3kl
2023-01-11T22:03:10.2952953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplyrcy3kl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2953224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2953334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2953612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2953748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2953931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxyr2n7y
2023-01-11T22:03:10.2954126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxyr2n7y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2954326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2954519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2954683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2954832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2954901Z ok (2.642s)
2023-01-11T22:03:10.2954905Z 
2023-01-11T22:03:10.2955104Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2955180Z Ran 1 test in 2.642s
2023-01-11T22:03:10.2955186Z 
2023-01-11T22:03:10.2955248Z OK
2023-01-11T22:03:10.2955253Z 
2023-01-11T22:03:10.2955336Z Generating XML reports...
2023-01-11T22:03:10.2955676Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214940.xml
2023-01-11T22:03:10.2955951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2956077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2956342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2956477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2956663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9qziauw
2023-01-11T22:03:10.2956857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9qziauw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2957111Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2957116Z 
2023-01-11T22:03:10.2957189Z Running tests...
2023-01-11T22:03:10.2957384Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2957612Z   test_py_class_constructor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28716
2023-01-11T22:03:10.2957759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28717
2023-01-11T22:03:10.2957916Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28718
2023-01-11T22:03:10.2958071Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28719
2023-01-11T22:03:10.2958346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2958470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2958753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2958976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2959166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj39rjaf_
2023-01-11T22:03:10.2959362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj39rjaf_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2959623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2959747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2960027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2960163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2960347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52ry7nww
2023-01-11T22:03:10.2960542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52ry7nww/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2960814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2960978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2961272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2961414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2961597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82pea19d
2023-01-11T22:03:10.2961791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82pea19d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2962067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2962191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2962472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2962610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2962796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10or79j9
2023-01-11T22:03:10.2962975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10or79j9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2963139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2963303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2963467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2963630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2963701Z ok (2.642s)
2023-01-11T22:03:10.2963706Z 
2023-01-11T22:03:10.2963903Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2963982Z Ran 1 test in 2.642s
2023-01-11T22:03:10.2963987Z 
2023-01-11T22:03:10.2964035Z OK
2023-01-11T22:03:10.2964053Z 
2023-01-11T22:03:10.2964127Z Generating XML reports...
2023-01-11T22:03:10.2964466Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214945.xml
2023-01-11T22:03:10.2964741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2964866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2965150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2965287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2965471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcymje90u
2023-01-11T22:03:10.2965668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcymje90u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2965911Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2965931Z 
2023-01-11T22:03:10.2965993Z Running tests...
2023-01-11T22:03:10.2966190Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2966421Z   test_py_class_instance_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28939
2023-01-11T22:03:10.2966580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28940
2023-01-11T22:03:10.2966737Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28941
2023-01-11T22:03:10.2966890Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28942
2023-01-11T22:03:10.2967167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2967325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2967625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2967764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2967949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38wmtv0s
2023-01-11T22:03:10.2968146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38wmtv0s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2968420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2968546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2968830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2968968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2969141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpak6vdbsi
2023-01-11T22:03:10.2969340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpak6vdbsi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2969614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2969742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2970023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2970161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2970348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzd61lfq9
2023-01-11T22:03:10.2970544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzd61lfq9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2970817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2970930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2971211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2971346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2971531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpruawbt7r
2023-01-11T22:03:10.2971727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpruawbt7r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2971894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2972059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2972224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2972376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2972446Z ok (2.637s)
2023-01-11T22:03:10.2972451Z 
2023-01-11T22:03:10.2972654Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2972729Z Ran 1 test in 2.637s
2023-01-11T22:03:10.2972735Z 
2023-01-11T22:03:10.2972796Z OK
2023-01-11T22:03:10.2972802Z 
2023-01-11T22:03:10.2972886Z Generating XML reports...
2023-01-11T22:03:10.2973229Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214950.xml
2023-01-11T22:03:10.2973504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2973629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2973931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2974099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2974286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5lyezwlj
2023-01-11T22:03:10.2974484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5lyezwlj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2974741Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2974747Z 
2023-01-11T22:03:10.2974822Z Running tests...
2023-01-11T22:03:10.2975020Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2975244Z   test_py_class_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29162
2023-01-11T22:03:10.2975387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29163
2023-01-11T22:03:10.2975545Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29164
2023-01-11T22:03:10.2975707Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29165
2023-01-11T22:03:10.2975983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2976108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2976390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2976525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2976713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8lu3klh
2023-01-11T22:03:10.2976909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8lu3klh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2977170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2977297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2977581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2977716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2977902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdf61ng5
2023-01-11T22:03:10.2978097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdf61ng5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2978400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2978525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2978792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2978929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2979114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjj_5g9lq
2023-01-11T22:03:10.2979308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjj_5g9lq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2979581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2979705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2979984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2980119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2980302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_dxcoft
2023-01-11T22:03:10.2980514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_dxcoft/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2980704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2980872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2981036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2981199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2981268Z ok (2.653s)
2023-01-11T22:03:10.2981273Z 
2023-01-11T22:03:10.2981470Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2981547Z Ran 1 test in 2.653s
2023-01-11T22:03:10.2981552Z 
2023-01-11T22:03:10.2981600Z OK
2023-01-11T22:03:10.2981619Z 
2023-01-11T22:03:10.2981794Z Generating XML reports...
2023-01-11T22:03:10.2982141Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214954.xml
2023-01-11T22:03:10.2982420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2982546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2982828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2982967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2983153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8y6ly23p
2023-01-11T22:03:10.2983350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8y6ly23p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2983623Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2983633Z 
2023-01-11T22:03:10.2983737Z Running tests...
2023-01-11T22:03:10.2983969Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2984199Z   test_py_class_static_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29385
2023-01-11T22:03:10.2984357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29386
2023-01-11T22:03:10.2984516Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29387
2023-01-11T22:03:10.2984668Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29388
2023-01-11T22:03:10.2984946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2985073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2985343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2985482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2985673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt82g41p4
2023-01-11T22:03:10.2985867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt82g41p4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2986142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2986268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2986550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2986686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2986856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpou59vrsf
2023-01-11T22:03:10.2987052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpou59vrsf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2987444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2987571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2987858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2987995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2988179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1kow1cq5
2023-01-11T22:03:10.2988375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1kow1cq5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2988647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2988757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2989040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2989176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2989361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmz43m06c
2023-01-11T22:03:10.2989559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmz43m06c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2989724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2989888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2990052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2990202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2990276Z ok (2.742s)
2023-01-11T22:03:10.2990281Z 
2023-01-11T22:03:10.2990478Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2990555Z Ran 1 test in 2.742s
2023-01-11T22:03:10.2990563Z 
2023-01-11T22:03:10.2990629Z OK
2023-01-11T22:03:10.2990634Z 
2023-01-11T22:03:10.2990720Z Generating XML reports...
2023-01-11T22:03:10.2991060Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214959.xml
2023-01-11T22:03:10.2991333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2991457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2991725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2991860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2992048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13dt48rv
2023-01-11T22:03:10.2992244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13dt48rv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2992504Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.2992509Z 
2023-01-11T22:03:10.2992583Z Running tests...
2023-01-11T22:03:10.2992779Z ----------------------------------------------------------------------
2023-01-11T22:03:10.2993007Z   test_py_function_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29608
2023-01-11T22:03:10.2993151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29609
2023-01-11T22:03:10.2993308Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29610
2023-01-11T22:03:10.2993461Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29611
2023-01-11T22:03:10.2993778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2993933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2994218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2994355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2994541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngak6jdc
2023-01-11T22:03:10.2994737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngak6jdc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2994993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2995117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2995399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2995538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2995726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdti6lvke
2023-01-11T22:03:10.2995920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdti6lvke/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2996194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2996319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2996585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2996719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2996903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp299ugsi
2023-01-11T22:03:10.2997098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp299ugsi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2997370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.2997494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.2997775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.2997910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.2998095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkw1jkbxy
2023-01-11T22:03:10.2998276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkw1jkbxy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.2998439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.2998607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.2998771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.2999006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.2999104Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.2999300Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2023-01-11T22:03:10.2999395Z Traceback (most recent call last):
2023-01-11T22:03:10.2999647Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.2999783Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.2999915Z TypeError: no_result() takes 0 positional arguments but 1 was given
2023-01-11T22:03:10.2999920Z 
2023-01-11T22:03:10.3000012Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3000245Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2023-01-11T22:03:10.3000338Z Traceback (most recent call last):
2023-01-11T22:03:10.3000671Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3000810Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3000926Z TypeError: no_result() takes 0 positional arguments but 1 was given
2023-01-11T22:03:10.3000945Z 
2023-01-11T22:03:10.3001023Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3001213Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2023-01-11T22:03:10.3001305Z Traceback (most recent call last):
2023-01-11T22:03:10.3001565Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3001697Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3001827Z TypeError: no_result() takes 0 positional arguments but 1 was given
2023-01-11T22:03:10.3001834Z 
2023-01-11T22:03:10.3001926Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3002107Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2023-01-11T22:03:10.3002201Z Traceback (most recent call last):
2023-01-11T22:03:10.3002465Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3002599Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3002730Z TypeError: no_result() takes 0 positional arguments but 1 was given
2023-01-11T22:03:10.3002736Z 
2023-01-11T22:03:10.3002806Z ok (2.752s)
2023-01-11T22:03:10.3002811Z 
2023-01-11T22:03:10.3003012Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3003089Z Ran 1 test in 2.752s
2023-01-11T22:03:10.3003095Z 
2023-01-11T22:03:10.3003156Z OK
2023-01-11T22:03:10.3003161Z 
2023-01-11T22:03:10.3003235Z Generating XML reports...
2023-01-11T22:03:10.3003576Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215003.xml
2023-01-11T22:03:10.3003854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3003980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3004264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3004402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3004591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpup22ihvq
2023-01-11T22:03:10.3004791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpup22ihvq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3005037Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3005058Z 
2023-01-11T22:03:10.3005118Z Running tests...
2023-01-11T22:03:10.3005317Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3005544Z   test_py_multi_async_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29831
2023-01-11T22:03:10.3005702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29832
2023-01-11T22:03:10.3005859Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29833
2023-01-11T22:03:10.3006013Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29834
2023-01-11T22:03:10.3006291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3006415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3006685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3006853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3007065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8f7nlag7
2023-01-11T22:03:10.3007264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8f7nlag7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3007540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3007666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3007948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3008085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3008258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvdrv9dai
2023-01-11T22:03:10.3008457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvdrv9dai/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3008732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3008857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3009138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3009275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3009463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3tpb64e
2023-01-11T22:03:10.3009659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3tpb64e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3009930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3010043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3010327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3010462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3010647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8p83s37
2023-01-11T22:03:10.3010842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8p83s37/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3011009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3011174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3011331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3011482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3011553Z ok (2.744s)
2023-01-11T22:03:10.3011558Z 
2023-01-11T22:03:10.3011755Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3011833Z Ran 1 test in 2.744s
2023-01-11T22:03:10.3011839Z 
2023-01-11T22:03:10.3011900Z OK
2023-01-11T22:03:10.3011905Z 
2023-01-11T22:03:10.3011991Z Generating XML reports...
2023-01-11T22:03:10.3012330Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215008.xml
2023-01-11T22:03:10.3012608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3012732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3012999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3013134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3013352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_t1y90n
2023-01-11T22:03:10.3013572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_t1y90n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3013831Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3013836Z 
2023-01-11T22:03:10.3013910Z Running tests...
2023-01-11T22:03:10.3014107Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3014330Z   test_py_nested_pickle (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30054
2023-01-11T22:03:10.3014488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30055
2023-01-11T22:03:10.3014631Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30056
2023-01-11T22:03:10.3014785Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30057
2023-01-11T22:03:10.3015063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3015190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3015475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3015612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3015799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwou3oprj
2023-01-11T22:03:10.3015994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwou3oprj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3016256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3016380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3016665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3016803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3016991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzp94j3dt
2023-01-11T22:03:10.3017188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzp94j3dt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3017459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3017585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3017866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3017988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3018175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8o4g1ij
2023-01-11T22:03:10.3018372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8o4g1ij/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3018645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3018769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3019048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3019183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3019369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplm1mh_t4
2023-01-11T22:03:10.3019551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplm1mh_t4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3019716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3019910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3020102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3020264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3020334Z ok (2.752s)
2023-01-11T22:03:10.3020340Z 
2023-01-11T22:03:10.3020541Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3020617Z Ran 1 test in 2.752s
2023-01-11T22:03:10.3020623Z 
2023-01-11T22:03:10.3020684Z OK
2023-01-11T22:03:10.3020690Z 
2023-01-11T22:03:10.3020762Z Generating XML reports...
2023-01-11T22:03:10.3021102Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215013.xml
2023-01-11T22:03:10.3021380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3021506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3021901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3022039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3022228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplnc3ik4c
2023-01-11T22:03:10.3022426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplnc3ik4c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3022668Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3022688Z 
2023-01-11T22:03:10.3022749Z Running tests...
2023-01-11T22:03:10.3022949Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3023174Z   test_py_no_return_result (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30277
2023-01-11T22:03:10.3023334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30278
2023-01-11T22:03:10.3023494Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30279
2023-01-11T22:03:10.3023649Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30280
2023-01-11T22:03:10.3023927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3024052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3024325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3024461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3024649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjcjno2y
2023-01-11T22:03:10.3024850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjcjno2y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3025125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3025250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3025533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3025673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3025846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53dqlmkv
2023-01-11T22:03:10.3026042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53dqlmkv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3026315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3026495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3026816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3026955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3027142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbulzw9t2
2023-01-11T22:03:10.3027418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3027543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3027811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3027946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3028132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjw5jb3x
2023-01-11T22:03:10.3028332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbulzw9t2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3028526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjw5jb3x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3028694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3028858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3029022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3029172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3029241Z do nothing
2023-01-11T22:03:10.3029306Z do nothing
2023-01-11T22:03:10.3029371Z do nothing
2023-01-11T22:03:10.3029435Z do nothing
2023-01-11T22:03:10.3029500Z do nothing
2023-01-11T22:03:10.3029566Z do nothing
2023-01-11T22:03:10.3029617Z do nothing
2023-01-11T22:03:10.3029681Z do nothing
2023-01-11T22:03:10.3029748Z ok (2.645s)
2023-01-11T22:03:10.3029753Z 
2023-01-11T22:03:10.3029956Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3030034Z Ran 1 test in 2.645s
2023-01-11T22:03:10.3030040Z 
2023-01-11T22:03:10.3030101Z OK
2023-01-11T22:03:10.3030107Z 
2023-01-11T22:03:10.3030191Z Generating XML reports...
2023-01-11T22:03:10.3030533Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215018.xml
2023-01-11T22:03:10.3030795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3030921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3031203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3031343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3031530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvith_tt6
2023-01-11T22:03:10.3031729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvith_tt6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3031987Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3031992Z 
2023-01-11T22:03:10.3032068Z Running tests...
2023-01-11T22:03:10.3032252Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3032477Z   test_py_raise_in_user_func (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30500
2023-01-11T22:03:10.3032636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30501
2023-01-11T22:03:10.3032791Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30502
2023-01-11T22:03:10.3032980Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30503
2023-01-11T22:03:10.3033300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3033426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3033710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3033847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3034020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpdg3_vt
2023-01-11T22:03:10.3034216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpdg3_vt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3034490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3034616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3034896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3035034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3035219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrtpxslq
2023-01-11T22:03:10.3035417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrtpxslq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3035676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3035801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3036080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3036214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3036404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7wcgddd
2023-01-11T22:03:10.3036603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7wcgddd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3036879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3037003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3037281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3037403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3037588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnrwg5ey
2023-01-11T22:03:10.3037784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnrwg5ey/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3037952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3038119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3038283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3038447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3038624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.3038803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.3039051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.3039231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.3039538Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3039919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3040214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3040511Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3040581Z ok (2.751s)
2023-01-11T22:03:10.3040586Z 
2023-01-11T22:03:10.3040787Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3040863Z Ran 1 test in 2.752s
2023-01-11T22:03:10.3040869Z 
2023-01-11T22:03:10.3040918Z OK
2023-01-11T22:03:10.3040922Z 
2023-01-11T22:03:10.3041008Z Generating XML reports...
2023-01-11T22:03:10.3041349Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215022.xml
2023-01-11T22:03:10.3041629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3041757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3042038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3042176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3042361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_4no7gz
2023-01-11T22:03:10.3042543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_4no7gz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3042798Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3042804Z 
2023-01-11T22:03:10.3042878Z Running tests...
2023-01-11T22:03:10.3043080Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3043322Z   test_py_raise_in_user_func_escaped_str (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30735
2023-01-11T22:03:10.3043484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30736
2023-01-11T22:03:10.3043640Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30737
2023-01-11T22:03:10.3043796Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30738
2023-01-11T22:03:10.3044071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3044183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3044467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3044605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3044792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn6gykfnb
2023-01-11T22:03:10.3044990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn6gykfnb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3045266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3045390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3045673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3045794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3045978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsoucab4y
2023-01-11T22:03:10.3046173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsoucab4y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3046485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3046640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3046923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3047058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3047243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2k8haso
2023-01-11T22:03:10.3047439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2k8haso/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3047696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3047821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3048101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3048240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3048429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdlx2zbhs
2023-01-11T22:03:10.3048625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdlx2zbhs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3048789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3048959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3049122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3049271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3049364Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3049581Z ValueError('\nFirst line of error \n next line of error \n last line of error')
2023-01-11T22:03:10.3049679Z Traceback (most recent call last):
2023-01-11T22:03:10.3049945Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3050082Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3050382Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 417, in raise_func_escape
2023-01-11T22:03:10.3050485Z     raise ValueError(expected_err_escape)
2023-01-11T22:03:10.3050546Z ValueError: 
2023-01-11T22:03:10.3050623Z First line of error 
2023-01-11T22:03:10.3050703Z  next line of error 
2023-01-11T22:03:10.3050777Z  last line of error
2023-01-11T22:03:10.3050783Z 
2023-01-11T22:03:10.3050875Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3051084Z ValueError('\nFirst line of error \n next line of error \n last line of error')
2023-01-11T22:03:10.3051178Z Traceback (most recent call last):
2023-01-11T22:03:10.3051430Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3051565Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3051859Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 417, in raise_func_escape
2023-01-11T22:03:10.3051961Z     raise ValueError(expected_err_escape)
2023-01-11T22:03:10.3052033Z ValueError: 
2023-01-11T22:03:10.3052110Z First line of error 
2023-01-11T22:03:10.3052184Z  next line of error 
2023-01-11T22:03:10.3052243Z  last line of error
2023-01-11T22:03:10.3052248Z 
2023-01-11T22:03:10.3052341Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3052549Z ValueError('\nFirst line of error \n next line of error \n last line of error')
2023-01-11T22:03:10.3052645Z Traceback (most recent call last):
2023-01-11T22:03:10.3052934Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3053098Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3053392Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 417, in raise_func_escape
2023-01-11T22:03:10.3053494Z     raise ValueError(expected_err_escape)
2023-01-11T22:03:10.3053551Z ValueError: 
2023-01-11T22:03:10.3053628Z First line of error 
2023-01-11T22:03:10.3053703Z  next line of error 
2023-01-11T22:03:10.3053775Z  last line of error
2023-01-11T22:03:10.3053780Z 
2023-01-11T22:03:10.3053871Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3054076Z ValueError('\nFirst line of error \n next line of error \n last line of error')
2023-01-11T22:03:10.3054169Z Traceback (most recent call last):
2023-01-11T22:03:10.3054413Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3054547Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3054840Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 417, in raise_func_escape
2023-01-11T22:03:10.3054942Z     raise ValueError(expected_err_escape)
2023-01-11T22:03:10.3055012Z ValueError: 
2023-01-11T22:03:10.3055087Z First line of error 
2023-01-11T22:03:10.3055177Z  next line of error 
2023-01-11T22:03:10.3055280Z  last line of error
2023-01-11T22:03:10.3055286Z 
2023-01-11T22:03:10.3055369Z ok (2.650s)
2023-01-11T22:03:10.3055376Z 
2023-01-11T22:03:10.3055592Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3055665Z Ran 1 test in 2.650s
2023-01-11T22:03:10.3055670Z 
2023-01-11T22:03:10.3055733Z OK
2023-01-11T22:03:10.3055738Z 
2023-01-11T22:03:10.3055823Z Generating XML reports...
2023-01-11T22:03:10.3056275Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215027.xml
2023-01-11T22:03:10.3056561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3056686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3056957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3057094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3057282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqe03058b
2023-01-11T22:03:10.3057475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqe03058b/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3057731Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3057736Z 
2023-01-11T22:03:10.3057810Z Running tests...
2023-01-11T22:03:10.3058009Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3058232Z   test_py_rpc_rref_args (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30958
2023-01-11T22:03:10.3058392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30959
2023-01-11T22:03:10.3058535Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30960
2023-01-11T22:03:10.3058689Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30961
2023-01-11T22:03:10.3058966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3059092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3059376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3059565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3059752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuv3l_13f
2023-01-11T22:03:10.3059978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuv3l_13f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3060241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3060368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3060650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3060785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3060973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpczw77109
2023-01-11T22:03:10.3061167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpczw77109/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3061445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3061572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3061967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3062091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3062277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpglogup63
2023-01-11T22:03:10.3062475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpglogup63/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3062754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3062884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3063170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3063310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3063495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbi2mykkc
2023-01-11T22:03:10.3063693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbi2mykkc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3063847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3064014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3064178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3064343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3064413Z ok (2.745s)
2023-01-11T22:03:10.3064421Z 
2023-01-11T22:03:10.3064619Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3064696Z Ran 1 test in 2.745s
2023-01-11T22:03:10.3064701Z 
2023-01-11T22:03:10.3064767Z OK
2023-01-11T22:03:10.3064773Z 
2023-01-11T22:03:10.3064845Z Generating XML reports...
2023-01-11T22:03:10.3065188Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215031.xml
2023-01-11T22:03:10.3065464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3065591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3065873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3066009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3066194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmf_wm9w
2023-01-11T22:03:10.3066447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmf_wm9w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3066743Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3066749Z 
2023-01-11T22:03:10.3066811Z Running tests...
2023-01-11T22:03:10.3067038Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3067287Z   test_py_rref_args (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31181
2023-01-11T22:03:10.3067447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31182
2023-01-11T22:03:10.3067605Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31183
2023-01-11T22:03:10.3067761Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31184
2023-01-11T22:03:10.3068043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3068172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3068445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3068582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3068767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp18ffvx2k
2023-01-11T22:03:10.3068964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp18ffvx2k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3069237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3069362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3069645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3069786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3069974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw10d6pjy
2023-01-11T22:03:10.3070158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw10d6pjy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3070431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3070556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3070838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3070975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3071160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44o6n81k
2023-01-11T22:03:10.3071357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44o6n81k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3071630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3071743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3072024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3072160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3072346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wx8xv6_
2023-01-11T22:03:10.3072539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wx8xv6_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3072706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3072871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3073080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3073271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3073327Z ok (2.537s)
2023-01-11T22:03:10.3073332Z 
2023-01-11T22:03:10.3073532Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3073609Z Ran 1 test in 2.538s
2023-01-11T22:03:10.3073614Z 
2023-01-11T22:03:10.3073675Z OK
2023-01-11T22:03:10.3073680Z 
2023-01-11T22:03:10.3073766Z Generating XML reports...
2023-01-11T22:03:10.3074107Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215036.xml
2023-01-11T22:03:10.3074383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3074510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3074777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3074916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3075105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk84wr4vg
2023-01-11T22:03:10.3075302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk84wr4vg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3075560Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3075566Z 
2023-01-11T22:03:10.3075639Z Running tests...
2023-01-11T22:03:10.3075835Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3076065Z   test_py_rref_args_user_share (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31404
2023-01-11T22:03:10.3076226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31405
2023-01-11T22:03:10.3076369Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31406
2023-01-11T22:03:10.3076524Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31407
2023-01-11T22:03:10.3076799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3076924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3077210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3077346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3077534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptadeq1ou
2023-01-11T22:03:10.3077732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptadeq1ou/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3077996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3078123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3078403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3078538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3078723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptsqrnbum
2023-01-11T22:03:10.3079005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptsqrnbum/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3079280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3079406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3079729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3079894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3080084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpue7qv1v7
2023-01-11T22:03:10.3080279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpue7qv1v7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3080553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3080679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3080959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3081093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3081281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2d5g87_t
2023-01-11T22:03:10.3081462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2d5g87_t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3081628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3081793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3081956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3082120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3082191Z ok (2.747s)
2023-01-11T22:03:10.3082197Z 
2023-01-11T22:03:10.3082396Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3082472Z Ran 1 test in 2.747s
2023-01-11T22:03:10.3082478Z 
2023-01-11T22:03:10.3082539Z OK
2023-01-11T22:03:10.3082545Z 
2023-01-11T22:03:10.3082619Z Generating XML reports...
2023-01-11T22:03:10.3082958Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215041.xml
2023-01-11T22:03:10.3083233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3083357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3083637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3083772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3083958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8p9cj_eh
2023-01-11T22:03:10.3084154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8p9cj_eh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3084394Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3084416Z 
2023-01-11T22:03:10.3084476Z Running tests...
2023-01-11T22:03:10.3084676Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3084892Z   test_py_tensors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31627
2023-01-11T22:03:10.3085050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31628
2023-01-11T22:03:10.3085206Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31629
2023-01-11T22:03:10.3085361Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31630
2023-01-11T22:03:10.3085638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3085762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3086031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3086203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3086418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz9cgcoy0
2023-01-11T22:03:10.3086616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz9cgcoy0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3086894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3087018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3087298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3087435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3087618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5g4md9e6
2023-01-11T22:03:10.3087877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3088003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3088282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3088416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3088601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07wl_46l
2023-01-11T22:03:10.3088796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5g4md9e6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3088986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07wl_46l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3089255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3089368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3089649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3089784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3089968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpalu5nned
2023-01-11T22:03:10.3090163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpalu5nned/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3090329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3090493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3090656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3090818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3090875Z ok (2.648s)
2023-01-11T22:03:10.3090881Z 
2023-01-11T22:03:10.3091079Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3091157Z Ran 1 test in 2.648s
2023-01-11T22:03:10.3091163Z 
2023-01-11T22:03:10.3091226Z OK
2023-01-11T22:03:10.3091232Z 
2023-01-11T22:03:10.3091317Z Generating XML reports...
2023-01-11T22:03:10.3091657Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215045.xml
2023-01-11T22:03:10.3091932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3092057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3092324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3092459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3092675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5nkjhsvh
2023-01-11T22:03:10.3092899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5nkjhsvh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3093156Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3093162Z 
2023-01-11T22:03:10.3093235Z Running tests...
2023-01-11T22:03:10.3093432Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3093654Z   test_py_tensors_in_container (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31850
2023-01-11T22:03:10.3093816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31851
2023-01-11T22:03:10.3093972Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31852
2023-01-11T22:03:10.3094129Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31853
2023-01-11T22:03:10.3094413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3094536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3094820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3094958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3095146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuby702ip
2023-01-11T22:03:10.3095330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuby702ip/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3095607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3095731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3096016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3096155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3096341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9d3_1iy
2023-01-11T22:03:10.3096535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9d3_1iy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3096811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3096949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3097216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3097353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3097542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgef5df7n
2023-01-11T22:03:10.3097740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgef5df7n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3098015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3098140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3098422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3098559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3098733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qc9e0vi
2023-01-11T22:03:10.3098929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qc9e0vi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3099095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3099292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3099482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3099646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3099716Z ok (2.647s)
2023-01-11T22:03:10.3099722Z 
2023-01-11T22:03:10.3099925Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3099987Z Ran 1 test in 2.647s
2023-01-11T22:03:10.3100006Z 
2023-01-11T22:03:10.3100055Z OK
2023-01-11T22:03:10.3100060Z 
2023-01-11T22:03:10.3100147Z Generating XML reports...
2023-01-11T22:03:10.3100491Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215050.xml
2023-01-11T22:03:10.3100767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3100894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3101179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3101319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3101507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcc1aybf
2023-01-11T22:03:10.3101827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcc1aybf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3102087Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3102093Z 
2023-01-11T22:03:10.3102168Z Running tests...
2023-01-11T22:03:10.3102366Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3102600Z   test_py_tensors_multi_async_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32073
2023-01-11T22:03:10.3102763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32074
2023-01-11T22:03:10.3102921Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32075
2023-01-11T22:03:10.3103076Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32076
2023-01-11T22:03:10.3103350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3103463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3103745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3103883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3104069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjt_zvglb
2023-01-11T22:03:10.3104266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjt_zvglb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3104543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3104669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3104952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3105074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3105261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6hcbhno
2023-01-11T22:03:10.3105457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6hcbhno/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3105729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3105945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3106261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3106400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3106585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeenxu4jj
2023-01-11T22:03:10.3106779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeenxu4jj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3107043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3107167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3107448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3107585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3107767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vm2m4aq
2023-01-11T22:03:10.3107963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vm2m4aq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3108131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3108296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3108447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3108610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3108678Z ok (2.852s)
2023-01-11T22:03:10.3108684Z 
2023-01-11T22:03:10.3108880Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3108959Z Ran 1 test in 2.852s
2023-01-11T22:03:10.3108964Z 
2023-01-11T22:03:10.3109030Z OK
2023-01-11T22:03:10.3109035Z 
2023-01-11T22:03:10.3109118Z Generating XML reports...
2023-01-11T22:03:10.3109462Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215054.xml
2023-01-11T22:03:10.3109737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3109847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3110127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3110263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3110447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporw2dgod
2023-01-11T22:03:10.3110641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporw2dgod/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3110898Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3110904Z 
2023-01-11T22:03:10.3110978Z Running tests...
2023-01-11T22:03:10.3111173Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3111377Z   test_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32296
2023-01-11T22:03:10.3111534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32297
2023-01-11T22:03:10.3111689Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32298
2023-01-11T22:03:10.3111843Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32299
2023-01-11T22:03:10.3112119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3112275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3112558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3112719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3112908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2r2xkrf
2023-01-11T22:03:10.3113092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2r2xkrf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3113365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3113488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3113768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3113903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3114091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobnvt7ar
2023-01-11T22:03:10.3114289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobnvt7ar/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3114560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3114682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3114949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3115082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3115267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6qax5b4
2023-01-11T22:03:10.3115460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6qax5b4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3115731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3115856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3116136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3116270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3116443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzso2c1wu
2023-01-11T22:03:10.3116638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzso2c1wu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3116801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3116964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3117125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3117288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3117355Z ok (2.646s)
2023-01-11T22:03:10.3117363Z 
2023-01-11T22:03:10.3117560Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3117623Z Ran 1 test in 2.646s
2023-01-11T22:03:10.3117641Z 
2023-01-11T22:03:10.3117690Z OK
2023-01-11T22:03:10.3117694Z 
2023-01-11T22:03:10.3117778Z Generating XML reports...
2023-01-11T22:03:10.3118120Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215059.xml
2023-01-11T22:03:10.3118394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3118517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3118795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3119056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3119271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0jwqbg1y
2023-01-11T22:03:10.3119456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0jwqbg1y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3119712Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3119718Z 
2023-01-11T22:03:10.3119793Z Running tests...
2023-01-11T22:03:10.3119994Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3120218Z   test_py_user_defined (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32519
2023-01-11T22:03:10.3120377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32520
2023-01-11T22:03:10.3120537Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32521
2023-01-11T22:03:10.3120692Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32522
2023-01-11T22:03:10.3120969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3121079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3121363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3121498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3121688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffylqphz
2023-01-11T22:03:10.3121960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3122085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3122366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3122503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3122676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mfty0ly
2023-01-11T22:03:10.3122874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffylqphz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3123065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mfty0ly/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3123338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3123460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3123739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3123875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3124061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2vbkszd
2023-01-11T22:03:10.3124253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2vbkszd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3124534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3124656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3124934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3125068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3125251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3d8bwgk
2023-01-11T22:03:10.3125445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3d8bwgk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3125654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3125857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3126008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3126169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3126237Z ok (2.647s)
2023-01-11T22:03:10.3126243Z 
2023-01-11T22:03:10.3126441Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3126517Z Ran 1 test in 2.648s
2023-01-11T22:03:10.3126523Z 
2023-01-11T22:03:10.3126583Z OK
2023-01-11T22:03:10.3126588Z 
2023-01-11T22:03:10.3126671Z Generating XML reports...
2023-01-11T22:03:10.3127010Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215104.xml
2023-01-11T22:03:10.3127285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3127399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3127681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3127816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3128001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8vrl3r4
2023-01-11T22:03:10.3128195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8vrl3r4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3128448Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3128454Z 
2023-01-11T22:03:10.3128526Z Running tests...
2023-01-11T22:03:10.3128726Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3128970Z   test_register_rpc_backend_and_set_and_start_rpc_backend (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32742
2023-01-11T22:03:10.3129126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32743
2023-01-11T22:03:10.3129280Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32744
2023-01-11T22:03:10.3129436Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32745
2023-01-11T22:03:10.3129708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3129834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3130116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3130254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3130438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6lkmie1n
2023-01-11T22:03:10.3130621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6lkmie1n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3130894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3131016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3131298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3131433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3131616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpooj5839c
2023-01-11T22:03:10.3131810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpooj5839c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3132115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3132265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3132534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3132668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3132850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5kb04zb
2023-01-11T22:03:10.3133042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5kb04zb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3133313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3133435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3133717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3133855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3134027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp36s05r4
2023-01-11T22:03:10.3134218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp36s05r4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3134380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3134544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3134709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3134869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3134936Z ok (2.446s)
2023-01-11T22:03:10.3134944Z 
2023-01-11T22:03:10.3135140Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3135214Z Ran 1 test in 2.447s
2023-01-11T22:03:10.3135219Z 
2023-01-11T22:03:10.3135270Z OK
2023-01-11T22:03:10.3135275Z 
2023-01-11T22:03:10.3135358Z Generating XML reports...
2023-01-11T22:03:10.3135700Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215108.xml
2023-01-11T22:03:10.3135976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3136101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3136381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3136514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3136697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8qsbie2
2023-01-11T22:03:10.3136881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8qsbie2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3137139Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3137144Z 
2023-01-11T22:03:10.3137218Z Running tests...
2023-01-11T22:03:10.3137413Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3137620Z   test_reinit (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 333
2023-01-11T22:03:10.3137773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 334
2023-01-11T22:03:10.3137925Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 335
2023-01-11T22:03:10.3138074Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 336
2023-01-11T22:03:10.3138350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3138497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3138805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3138941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3139129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5uyp488
2023-01-11T22:03:10.3139324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5uyp488/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3139597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3139721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3140001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3140126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3140311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30tvgtfn
2023-01-11T22:03:10.3140504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30tvgtfn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3140777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3140899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3141178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3141313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3141496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_gjfsz3
2023-01-11T22:03:10.3141822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_gjfsz3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3142088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3142212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3142490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3142626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3142809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwm_obzhs
2023-01-11T22:03:10.3143005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwm_obzhs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3143171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3143336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3143489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3143653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3143832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.3144012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.3144187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.3144490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3144662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.3144962Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3145352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3145647Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3145738Z ok (2.752s)
2023-01-11T22:03:10.3145743Z 
2023-01-11T22:03:10.3145943Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3146022Z Ran 1 test in 2.752s
2023-01-11T22:03:10.3146027Z 
2023-01-11T22:03:10.3146090Z OK
2023-01-11T22:03:10.3146095Z 
2023-01-11T22:03:10.3146181Z Generating XML reports...
2023-01-11T22:03:10.3146521Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215113.xml
2023-01-11T22:03:10.3146797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3146925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3147198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3147336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3147522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqyp8tt1a
2023-01-11T22:03:10.3147718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqyp8tt1a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3147974Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3147979Z 
2023-01-11T22:03:10.3148051Z Running tests...
2023-01-11T22:03:10.3148247Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3148469Z   test_remote_same_worker (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 568
2023-01-11T22:03:10.3148625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 569
2023-01-11T22:03:10.3148766Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 570
2023-01-11T22:03:10.3148918Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 571
2023-01-11T22:03:10.3149194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3149318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3149599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3149734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3149921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0g9zjjce
2023-01-11T22:03:10.3150123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0g9zjjce/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3150384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3150510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3150792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3150927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3151113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp118jn3uh
2023-01-11T22:03:10.3151308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp118jn3uh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3151584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3151752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3152061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3152184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3152369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6wi6imz9
2023-01-11T22:03:10.3152565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6wi6imz9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3152837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3152961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3153243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3153379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3153573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeu06_szs
2023-01-11T22:03:10.3153755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeu06_szs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3153921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3154087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3154250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3154412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3154480Z ok (2.743s)
2023-01-11T22:03:10.3154486Z 
2023-01-11T22:03:10.3154685Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3154762Z Ran 1 test in 2.744s
2023-01-11T22:03:10.3154770Z 
2023-01-11T22:03:10.3154833Z OK
2023-01-11T22:03:10.3154838Z 
2023-01-11T22:03:10.3154911Z Generating XML reports...
2023-01-11T22:03:10.3155252Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215117.xml
2023-01-11T22:03:10.3155526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3155651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3155933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3156067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3156254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd6au6cbz
2023-01-11T22:03:10.3156449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd6au6cbz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3156692Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3156711Z 
2023-01-11T22:03:10.3156771Z Running tests...
2023-01-11T22:03:10.3156970Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3157185Z   test_remote_throw (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 793
2023-01-11T22:03:10.3157341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 794
2023-01-11T22:03:10.3157493Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 795
2023-01-11T22:03:10.3157646Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 796
2023-01-11T22:03:10.3157923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3158048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3158350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3158515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3158704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpndb7sruv
2023-01-11T22:03:10.3158978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpndb7sruv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3159258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3159383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3159668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3159808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3159981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3tiyofc
2023-01-11T22:03:10.3160176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3tiyofc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3160454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3160579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3160864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3161001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3161184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_m4bh3xv
2023-01-11T22:03:10.3161380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_m4bh3xv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3161654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3161767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3162046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3162180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3162365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoe_7xwn6
2023-01-11T22:03:10.3162556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoe_7xwn6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3162720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3162885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3163046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3163210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3163291Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3163415Z ValueError('Expected error')
2023-01-11T22:03:10.3163508Z Traceback (most recent call last):
2023-01-11T22:03:10.3163770Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3163905Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3164193Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc
2023-01-11T22:03:10.3164290Z     raise ValueError("Expected error")
2023-01-11T22:03:10.3164364Z ValueError: Expected error
2023-01-11T22:03:10.3164381Z 
2023-01-11T22:03:10.3164459Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3164578Z ValueError('Expected error')
2023-01-11T22:03:10.3164669Z Traceback (most recent call last):
2023-01-11T22:03:10.3164967Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3165129Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3165417Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc
2023-01-11T22:03:10.3165514Z     raise ValueError("Expected error")
2023-01-11T22:03:10.3165586Z ValueError: Expected error
2023-01-11T22:03:10.3165592Z 
2023-01-11T22:03:10.3165680Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3165798Z ValueError('Expected error')
2023-01-11T22:03:10.3165888Z Traceback (most recent call last):
2023-01-11T22:03:10.3166145Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3166278Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3166562Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc
2023-01-11T22:03:10.3166663Z     raise ValueError("Expected error")
2023-01-11T22:03:10.3166738Z ValueError: Expected error
2023-01-11T22:03:10.3166744Z 
2023-01-11T22:03:10.3166833Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3166952Z ValueError('Expected error')
2023-01-11T22:03:10.3167043Z Traceback (most recent call last):
2023-01-11T22:03:10.3167299Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3167432Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3167720Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc
2023-01-11T22:03:10.3167813Z     raise ValueError("Expected error")
2023-01-11T22:03:10.3167886Z ValueError: Expected error
2023-01-11T22:03:10.3167891Z 
2023-01-11T22:03:10.3167960Z ok (2.647s)
2023-01-11T22:03:10.3167965Z 
2023-01-11T22:03:10.3168163Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3168239Z Ran 1 test in 2.648s
2023-01-11T22:03:10.3168246Z 
2023-01-11T22:03:10.3168307Z OK
2023-01-11T22:03:10.3168312Z 
2023-01-11T22:03:10.3168395Z Generating XML reports...
2023-01-11T22:03:10.3168737Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215122.xml
2023-01-11T22:03:10.3169011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3169123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3169405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3169542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3169730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65kj4mmj
2023-01-11T22:03:10.3169928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65kj4mmj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3170182Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3170187Z 
2023-01-11T22:03:10.3170259Z Running tests...
2023-01-11T22:03:10.3170455Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3170681Z   test_remote_with_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1016
2023-01-11T22:03:10.3170823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1017
2023-01-11T22:03:10.3170976Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1018
2023-01-11T22:03:10.3171129Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1019
2023-01-11T22:03:10.3171437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3171602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3171886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3172022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3172208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppxk_l7p6
2023-01-11T22:03:10.3172391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppxk_l7p6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3172662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3172790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3173067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3173203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3173389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_6yv_a_
2023-01-11T22:03:10.3173581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_6yv_a_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3173854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3173977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3174245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3174379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3174564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7nri52t
2023-01-11T22:03:10.3174763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7nri52t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3175038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3175163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3175441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3175576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3175749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajj465fp
2023-01-11T22:03:10.3175949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajj465fp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3176116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3176277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3176443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3176605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3176697Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3176889Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2023-01-11T22:03:10.3176983Z Traceback (most recent call last):
2023-01-11T22:03:10.3177234Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3177368Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3177499Z TypeError: no_result() takes 0 positional arguments but 1 was given
2023-01-11T22:03:10.3177504Z 
2023-01-11T22:03:10.3177594Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3177748Z ValueError('Expected error')
2023-01-11T22:03:10.3177840Z Traceback (most recent call last):
2023-01-11T22:03:10.3178123Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3178242Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3178527Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3178619Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3178705Z ValueError: Expected error
2023-01-11T22:03:10.3178711Z 
2023-01-11T22:03:10.3178801Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3178923Z ValueError('Expected error')
2023-01-11T22:03:10.3179015Z Traceback (most recent call last):
2023-01-11T22:03:10.3179274Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3179393Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3179677Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3179771Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3179857Z ValueError: Expected error
2023-01-11T22:03:10.3179862Z 
2023-01-11T22:03:10.3179952Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3180143Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2023-01-11T22:03:10.3180232Z Traceback (most recent call last):
2023-01-11T22:03:10.3180490Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3180609Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3180737Z TypeError: no_result() takes 0 positional arguments but 1 was given
2023-01-11T22:03:10.3180742Z 
2023-01-11T22:03:10.3180831Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3180954Z ValueError('Expected error')
2023-01-11T22:03:10.3181047Z Traceback (most recent call last):
2023-01-11T22:03:10.3181305Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3181436Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3182052Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3182167Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3182269Z ValueError: Expected error
2023-01-11T22:03:10.3182275Z 
2023-01-11T22:03:10.3182366Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3182560Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2023-01-11T22:03:10.3182653Z Traceback (most recent call last):
2023-01-11T22:03:10.3182910Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3183046Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3183178Z TypeError: no_result() takes 0 positional arguments but 1 was given
2023-01-11T22:03:10.3183183Z 
2023-01-11T22:03:10.3183262Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3183385Z ValueError('Expected error')
2023-01-11T22:03:10.3183477Z Traceback (most recent call last):
2023-01-11T22:03:10.3183735Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3183869Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3184151Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3184245Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3184331Z ValueError: Expected error
2023-01-11T22:03:10.3184336Z 
2023-01-11T22:03:10.3184412Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3184677Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2023-01-11T22:03:10.3184767Z Traceback (most recent call last):
2023-01-11T22:03:10.3185064Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3185199Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3185332Z TypeError: no_result() takes 0 positional arguments but 1 was given
2023-01-11T22:03:10.3185337Z 
2023-01-11T22:03:10.3185407Z ok (2.755s)
2023-01-11T22:03:10.3185413Z 
2023-01-11T22:03:10.3185613Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3185676Z Ran 1 test in 2.755s
2023-01-11T22:03:10.3185682Z 
2023-01-11T22:03:10.3185746Z OK
2023-01-11T22:03:10.3185751Z 
2023-01-11T22:03:10.3185835Z Generating XML reports...
2023-01-11T22:03:10.3186182Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215127.xml
2023-01-11T22:03:10.3186465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3186594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3186878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3187015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3187201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9mfl7ej
2023-01-11T22:03:10.3187389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9mfl7ej/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3187644Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3187649Z 
2023-01-11T22:03:10.3187723Z Running tests...
2023-01-11T22:03:10.3187921Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3188142Z   test_return_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1239
2023-01-11T22:03:10.3188302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1240
2023-01-11T22:03:10.3188453Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1241
2023-01-11T22:03:10.3188599Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1242
2023-01-11T22:03:10.3188863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3188987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3189270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3189406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3189595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfoved54k
2023-01-11T22:03:10.3189794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfoved54k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3190069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3190194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3190476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3190599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3190783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3t0epgdq
2023-01-11T22:03:10.3190981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3t0epgdq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3191253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3191410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3191721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3191858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3192043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaq1xgbjm
2023-01-11T22:03:10.3192226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaq1xgbjm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3192495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3192619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3192902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3193040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3193226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1g6ciyag
2023-01-11T22:03:10.3193421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1g6ciyag/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3193585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3193749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3193899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3194062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3194131Z ok (2.645s)
2023-01-11T22:03:10.3194138Z 
2023-01-11T22:03:10.3194337Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3194415Z Ran 1 test in 2.645s
2023-01-11T22:03:10.3194420Z 
2023-01-11T22:03:10.3194480Z OK
2023-01-11T22:03:10.3194485Z 
2023-01-11T22:03:10.3194570Z Generating XML reports...
2023-01-11T22:03:10.3194913Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215131.xml
2023-01-11T22:03:10.3195176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3195299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3195582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3195717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3195900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfs_k162y
2023-01-11T22:03:10.3196099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfs_k162y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3196360Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3196366Z 
2023-01-11T22:03:10.3196438Z Running tests...
2023-01-11T22:03:10.3196635Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3196845Z   test_return_future_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1462
2023-01-11T22:03:10.3196999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1463
2023-01-11T22:03:10.3197156Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1464
2023-01-11T22:03:10.3197306Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1465
2023-01-11T22:03:10.3197584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3197743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3198055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3198192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3198367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9yw7ze58
2023-01-11T22:03:10.3198560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9yw7ze58/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3198939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3199066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3199352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3199492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3199678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7_b3yyw
2023-01-11T22:03:10.3199875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7_b3yyw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3200149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3200260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3200542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3200678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3200865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5vysdyd
2023-01-11T22:03:10.3201060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5vysdyd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3201336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3201464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3201747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3201883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3202056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpequ3ohoy
2023-01-11T22:03:10.3202253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpequ3ohoy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3202420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3202585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3202751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3202918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3202988Z ok (2.750s)
2023-01-11T22:03:10.3202993Z 
2023-01-11T22:03:10.3203195Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3203258Z Ran 1 test in 2.751s
2023-01-11T22:03:10.3203263Z 
2023-01-11T22:03:10.3203326Z OK
2023-01-11T22:03:10.3203331Z 
2023-01-11T22:03:10.3203417Z Generating XML reports...
2023-01-11T22:03:10.3203761Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215136.xml
2023-01-11T22:03:10.3204037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3204163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3204482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3204690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3204877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02nacmku
2023-01-11T22:03:10.3205061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02nacmku/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3205320Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3205325Z 
2023-01-11T22:03:10.3205400Z Running tests...
2023-01-11T22:03:10.3205598Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3205821Z   test_return_future_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1685
2023-01-11T22:03:10.3205979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1686
2023-01-11T22:03:10.3206138Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1687
2023-01-11T22:03:10.3206294Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1688
2023-01-11T22:03:10.3206558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3206684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3206972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3207109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3207297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppxfy7o4i
2023-01-11T22:03:10.3207498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppxfy7o4i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3207777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3207904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3208188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3208312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3208501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyupct261
2023-01-11T22:03:10.3208698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyupct261/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3208971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3209096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3209378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3209518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3209706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpbynwu_
2023-01-11T22:03:10.3209889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpbynwu_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3210162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3210289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3210571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3210708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3210894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsf386wyi
2023-01-11T22:03:10.3211120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsf386wyi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3211311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3211480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3211631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3211792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3211861Z ok (2.747s)
2023-01-11T22:03:10.3211866Z 
2023-01-11T22:03:10.3212064Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3212140Z Ran 1 test in 2.747s
2023-01-11T22:03:10.3212146Z 
2023-01-11T22:03:10.3212208Z OK
2023-01-11T22:03:10.3212213Z 
2023-01-11T22:03:10.3212295Z Generating XML reports...
2023-01-11T22:03:10.3212638Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215141.xml
2023-01-11T22:03:10.3212907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3213034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3213316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3213451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3213634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjpyoqdc
2023-01-11T22:03:10.3213832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjpyoqdc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3214087Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3214092Z 
2023-01-11T22:03:10.3214167Z Running tests...
2023-01-11T22:03:10.3214365Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3214575Z   test_return_local_rrefs (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1908
2023-01-11T22:03:10.3214729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1909
2023-01-11T22:03:10.3214881Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1910
2023-01-11T22:03:10.3215033Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1911
2023-01-11T22:03:10.3215309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3215432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3215714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3215851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3216023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_f6mgpcf
2023-01-11T22:03:10.3216224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_f6mgpcf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3216499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3216621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3216901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3217035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3217221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpza6od76a
2023-01-11T22:03:10.3217416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpza6od76a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3217718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3217868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3218150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3218285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3218471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxoco5u0l
2023-01-11T22:03:10.3218666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxoco5u0l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3218935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3219057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3219339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3219475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3219646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiqytjjfg
2023-01-11T22:03:10.3219844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiqytjjfg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3220008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3220170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3220332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3220492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3220559Z ok (2.749s)
2023-01-11T22:03:10.3220567Z 
2023-01-11T22:03:10.3220766Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3220828Z Ran 1 test in 2.749s
2023-01-11T22:03:10.3220834Z 
2023-01-11T22:03:10.3220898Z OK
2023-01-11T22:03:10.3220904Z 
2023-01-11T22:03:10.3220989Z Generating XML reports...
2023-01-11T22:03:10.3221328Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215145.xml
2023-01-11T22:03:10.3221601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3221847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3222134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3222270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3222456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqeov6zud
2023-01-11T22:03:10.3222641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqeov6zud/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3222904Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3222910Z 
2023-01-11T22:03:10.3222984Z Running tests...
2023-01-11T22:03:10.3223183Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3223404Z   test_rpc_barrier_all (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2131
2023-01-11T22:03:10.3223564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2132
2023-01-11T22:03:10.3223718Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2133
2023-01-11T22:03:10.3223871Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2134
2023-01-11T22:03:10.3224190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3224317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3224634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3224773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3224957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_h51m1fz
2023-01-11T22:03:10.3225152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_h51m1fz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3225425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3225549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3225831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3225956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3226142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrmr43ui
2023-01-11T22:03:10.3226338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrmr43ui/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3226611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3226739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3227019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3227156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3227340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpom3ng7ob
2023-01-11T22:03:10.3227523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpom3ng7ob/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3227794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3227918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3228201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3228334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3228518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zwga7gl
2023-01-11T22:03:10.3228712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zwga7gl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3228876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3229042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3229191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3229353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3229420Z ok (2.650s)
2023-01-11T22:03:10.3229426Z 
2023-01-11T22:03:10.3229624Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3229700Z Ran 1 test in 2.650s
2023-01-11T22:03:10.3229705Z 
2023-01-11T22:03:10.3229764Z OK
2023-01-11T22:03:10.3229769Z 
2023-01-11T22:03:10.3229853Z Generating XML reports...
2023-01-11T22:03:10.3230189Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215150.xml
2023-01-11T22:03:10.3230449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3230609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3230919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3231055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3231240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9ir1vp0
2023-01-11T22:03:10.3231434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9ir1vp0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3231691Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3231697Z 
2023-01-11T22:03:10.3231770Z Running tests...
2023-01-11T22:03:10.3231964Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3232184Z   test_rpc_barrier_multithreaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2354
2023-01-11T22:03:10.3232343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2355
2023-01-11T22:03:10.3232498Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2356
2023-01-11T22:03:10.3232648Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2357
2023-01-11T22:03:10.3232924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3233049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3233332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3233467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3233641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6vyd20w
2023-01-11T22:03:10.3233835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6vyd20w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3234112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3234237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3234518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3234653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3234839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazsuxyti
2023-01-11T22:03:10.3235036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazsuxyti/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3235306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3235417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3235699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3235838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3236023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphnwq929l
2023-01-11T22:03:10.3236217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphnwq929l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3236488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3236610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3236887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3237020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3237224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjmx56cz
2023-01-11T22:03:10.3237446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjmx56cz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3237611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3237776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3237939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3238101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3238167Z ok (2.641s)
2023-01-11T22:03:10.3238173Z 
2023-01-11T22:03:10.3238371Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3238433Z Ran 1 test in 2.641s
2023-01-11T22:03:10.3238438Z 
2023-01-11T22:03:10.3238497Z OK
2023-01-11T22:03:10.3238505Z 
2023-01-11T22:03:10.3238592Z Generating XML reports...
2023-01-11T22:03:10.3239032Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215155.xml
2023-01-11T22:03:10.3239315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3239440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3239728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3239864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3240051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3dkrjx6
2023-01-11T22:03:10.3240235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3dkrjx6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3240493Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3240500Z 
2023-01-11T22:03:10.3240575Z Running tests...
2023-01-11T22:03:10.3240778Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3241012Z   test_rpc_barrier_partial_subset (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2589
2023-01-11T22:03:10.3241168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2590
2023-01-11T22:03:10.3241324Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2591
2023-01-11T22:03:10.3241478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2592
2023-01-11T22:03:10.3241741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3241866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3242148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3242287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3242476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptwcyk7fv
2023-01-11T22:03:10.3242674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptwcyk7fv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3242947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3243074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3243353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3243476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3243657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_komoaf2
2023-01-11T22:03:10.3243899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_komoaf2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3244204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3244329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3244611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3244748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3244934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkg2zzfce
2023-01-11T22:03:10.3245119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkg2zzfce/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3245389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3245516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3245807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3245947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3246137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi01lilha
2023-01-11T22:03:10.3246334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi01lilha/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3246500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3246665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3246812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3246974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3247078Z ok (2.750s)
2023-01-11T22:03:10.3247086Z 
2023-01-11T22:03:10.3247290Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3247364Z Ran 1 test in 2.750s
2023-01-11T22:03:10.3247370Z 
2023-01-11T22:03:10.3247434Z OK
2023-01-11T22:03:10.3247440Z 
2023-01-11T22:03:10.3247523Z Generating XML reports...
2023-01-11T22:03:10.3247866Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215159.xml
2023-01-11T22:03:10.3248130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3248254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3248536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3248674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3248860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0yi9ev2
2023-01-11T22:03:10.3249058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0yi9ev2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3249318Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3249323Z 
2023-01-11T22:03:10.3249396Z Running tests...
2023-01-11T22:03:10.3249594Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3249805Z   test_rpc_barrier_subset (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2812
2023-01-11T22:03:10.3249962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2813
2023-01-11T22:03:10.3250116Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2814
2023-01-11T22:03:10.3250303Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2815
2023-01-11T22:03:10.3250608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3250733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3251016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3251150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3251323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hutlxae
2023-01-11T22:03:10.3251519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hutlxae/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3251793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3251916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3252200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3252338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3252523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpflz_nfaj
2023-01-11T22:03:10.3252717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpflz_nfaj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3252991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3253102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3253383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3253517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3253704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9o74uk63
2023-01-11T22:03:10.3253898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9o74uk63/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3254171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3254293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3254573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3254706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3254877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8jy6utd
2023-01-11T22:03:10.3255071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8jy6utd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3255237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3255400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3255565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3255727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3255796Z ok (2.743s)
2023-01-11T22:03:10.3255802Z 
2023-01-11T22:03:10.3256004Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3256067Z Ran 1 test in 2.743s
2023-01-11T22:03:10.3256072Z 
2023-01-11T22:03:10.3256135Z OK
2023-01-11T22:03:10.3256140Z 
2023-01-11T22:03:10.3256229Z Generating XML reports...
2023-01-11T22:03:10.3256575Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215204.xml
2023-01-11T22:03:10.3256851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3257031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3257343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3257482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3257668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjabokpr4
2023-01-11T22:03:10.3257853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjabokpr4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3258115Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3258120Z 
2023-01-11T22:03:10.3258199Z Running tests...
2023-01-11T22:03:10.3258399Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3258637Z   test_rpc_profiling_async_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3035
2023-01-11T22:03:10.3258796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3036
2023-01-11T22:03:10.3258952Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3037
2023-01-11T22:03:10.3259104Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3038
2023-01-11T22:03:10.3259367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3259492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3259773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3259911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3260096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0til03r
2023-01-11T22:03:10.3260296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0til03r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3260572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3260699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3260979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3261103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3261289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuhi_gi3l
2023-01-11T22:03:10.3261485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuhi_gi3l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3261862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3261993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3262279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3262416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3262602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz68m_25k
2023-01-11T22:03:10.3262783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz68m_25k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3263056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3263181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3263461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3263648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3263882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3benw7x2
2023-01-11T22:03:10.3264080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3benw7x2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3264246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3264412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3264562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3264727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3264908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.3265087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.3265267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.3265575Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3265877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3266054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.3266356Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3266640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3266708Z ok (3.753s)
2023-01-11T22:03:10.3266715Z 
2023-01-11T22:03:10.3266917Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3266994Z Ran 1 test in 3.754s
2023-01-11T22:03:10.3266999Z 
2023-01-11T22:03:10.3267062Z OK
2023-01-11T22:03:10.3267069Z 
2023-01-11T22:03:10.3267156Z Generating XML reports...
2023-01-11T22:03:10.3267500Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215209.xml
2023-01-11T22:03:10.3267776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3267901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3268169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3268305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3268492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0cke6b21
2023-01-11T22:03:10.3268691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0cke6b21/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3268948Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3268954Z 
2023-01-11T22:03:10.3269026Z Running tests...
2023-01-11T22:03:10.3269224Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3269475Z   test_rpc_profiling_async_function_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3270
2023-01-11T22:03:10.3269619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3271
2023-01-11T22:03:10.3269771Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3272
2023-01-11T22:03:10.3269922Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3273
2023-01-11T22:03:10.3270229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3270353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3270666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3270804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3270990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsws1gra2
2023-01-11T22:03:10.3271184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsws1gra2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3271445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3271569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3271852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3271991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3272177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcbba8si
2023-01-11T22:03:10.3272449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3272572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3272853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3272975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3273158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6tznnzkv
2023-01-11T22:03:10.3273431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3273558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3273839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3273973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3274156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6a2_zqf
2023-01-11T22:03:10.3274351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcbba8si/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3274541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6tznnzkv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3274715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6a2_zqf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3274879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3275045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3275207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3275370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3275547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.3275724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.3276024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3276195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.3276481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3276686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.3277013Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3277313Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3277384Z ok (3.747s)
2023-01-11T22:03:10.3277390Z 
2023-01-11T22:03:10.3277586Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3277661Z Ran 1 test in 3.747s
2023-01-11T22:03:10.3277667Z 
2023-01-11T22:03:10.3277727Z OK
2023-01-11T22:03:10.3277733Z 
2023-01-11T22:03:10.3277817Z Generating XML reports...
2023-01-11T22:03:10.3278146Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215214.xml
2023-01-11T22:03:10.3278421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3278547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3278904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3279045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3279234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_0af9z_
2023-01-11T22:03:10.3279430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_0af9z_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3279688Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3279694Z 
2023-01-11T22:03:10.3279755Z Running tests...
2023-01-11T22:03:10.3279953Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3280197Z   test_rpc_profiling_remote_record_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3506
2023-01-11T22:03:10.3280358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3507
2023-01-11T22:03:10.3280513Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3508
2023-01-11T22:03:10.3280667Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3509
2023-01-11T22:03:10.3280941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3281067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3281350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3281473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3281660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4y5niu_
2023-01-11T22:03:10.3281860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4y5niu_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3282140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3282265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3282550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3282687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3282869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_2re2g5
2023-01-11T22:03:10.3283049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_2re2g5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3283319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3283477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3283785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3283922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3284106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1urv2zhh
2023-01-11T22:03:10.3284302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1urv2zhh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3284573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3284699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3284968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3285111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3285296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvticrimm
2023-01-11T22:03:10.3285490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvticrimm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3285655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3285818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3285980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3286139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3286194Z ok (2.739s)
2023-01-11T22:03:10.3286212Z 
2023-01-11T22:03:10.3286396Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3286473Z Ran 1 test in 2.740s
2023-01-11T22:03:10.3286479Z 
2023-01-11T22:03:10.3286601Z OK
2023-01-11T22:03:10.3286606Z 
2023-01-11T22:03:10.3286689Z Generating XML reports...
2023-01-11T22:03:10.3287031Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215220.xml
2023-01-11T22:03:10.3287304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3287428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3287708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3293172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3293401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoh6t_6l6
2023-01-11T22:03:10.3293608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoh6t_6l6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3293898Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3293906Z 
2023-01-11T22:03:10.3293984Z Running tests...
2023-01-11T22:03:10.3294187Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3294412Z   test_rpc_return_rref (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3729
2023-01-11T22:03:10.3294572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3730
2023-01-11T22:03:10.3294728Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3731
2023-01-11T22:03:10.3294883Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3732
2023-01-11T22:03:10.3295151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3295372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3295660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3295836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3296029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprk88ughb
2023-01-11T22:03:10.3296230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprk88ughb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3296508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3296633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3296904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3297041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3297228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ngb2n09
2023-01-11T22:03:10.3297425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ngb2n09/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3297699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3297824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3298108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3298244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3298430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkobu8g7m
2023-01-11T22:03:10.3298612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkobu8g7m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3298886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3299012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3299291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3299427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3299612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptx5mj98_
2023-01-11T22:03:10.3299805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptx5mj98_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3299970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3300135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3300286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3300449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3300516Z ok (2.642s)
2023-01-11T22:03:10.3300524Z 
2023-01-11T22:03:10.3300726Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3300802Z Ran 1 test in 2.643s
2023-01-11T22:03:10.3300808Z 
2023-01-11T22:03:10.3300870Z OK
2023-01-11T22:03:10.3300875Z 
2023-01-11T22:03:10.3300959Z Generating XML reports...
2023-01-11T22:03:10.3301300Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215225.xml
2023-01-11T22:03:10.3301562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3301820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3302103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3302303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3302525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8cefmo8b
2023-01-11T22:03:10.3302725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8cefmo8b/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3302984Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3302991Z 
2023-01-11T22:03:10.3303065Z Running tests...
2023-01-11T22:03:10.3303264Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3303470Z   test_rpc_timeouts (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3952
2023-01-11T22:03:10.3303628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3953
2023-01-11T22:03:10.3303784Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3954
2023-01-11T22:03:10.3303937Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3955
2023-01-11T22:03:10.3304215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3304341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3304624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3304760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3304959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_jf0o5a
2023-01-11T22:03:10.3305153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_jf0o5a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3305425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3305552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3305832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3305968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3306151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsq6fjs3z
2023-01-11T22:03:10.3306346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsq6fjs3z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3306618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3306728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3307007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3307144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3307332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5m30w05e
2023-01-11T22:03:10.3307525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5m30w05e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3307796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3307919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3308199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3308321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3308503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6nvf5a83
2023-01-11T22:03:10.3308697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6nvf5a83/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3308894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3309084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3309251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3309415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3309483Z ok (8.863s)
2023-01-11T22:03:10.3309488Z 
2023-01-11T22:03:10.3309687Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3309752Z Ran 1 test in 8.863s
2023-01-11T22:03:10.3309757Z 
2023-01-11T22:03:10.3309818Z OK
2023-01-11T22:03:10.3309822Z 
2023-01-11T22:03:10.3309908Z Generating XML reports...
2023-01-11T22:03:10.3310249Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215229.xml
2023-01-11T22:03:10.3310527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3310652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3310934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3311070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3311241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1r08enqk
2023-01-11T22:03:10.3311434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1r08enqk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3311689Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3311695Z 
2023-01-11T22:03:10.3311770Z Running tests...
2023-01-11T22:03:10.3311970Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3312200Z   test_rref_context_debug_info (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4175
2023-01-11T22:03:10.3312358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4176
2023-01-11T22:03:10.3312512Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4177
2023-01-11T22:03:10.3312666Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4178
2023-01-11T22:03:10.3312933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3313058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3313347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3313484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3313674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxftgovt
2023-01-11T22:03:10.3313873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxftgovt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3314146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3314270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3314543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3314679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3314863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qmowfl4
2023-01-11T22:03:10.3315059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qmowfl4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3315400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3315569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3315852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3315988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3316177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwqjrhxwp
2023-01-11T22:03:10.3316362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwqjrhxwp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3316633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3316757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3317042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3317182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3317371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxi8ppqzb
2023-01-11T22:03:10.3317566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxi8ppqzb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3317730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3317881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3318044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3318205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3318383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.3318568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.3318966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3319279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3319456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.3319632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.3319915Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3320212Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3320284Z ok (2.951s)
2023-01-11T22:03:10.3320290Z 
2023-01-11T22:03:10.3320488Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3320568Z Ran 1 test in 2.951s
2023-01-11T22:03:10.3320574Z 
2023-01-11T22:03:10.3320636Z OK
2023-01-11T22:03:10.3320641Z 
2023-01-11T22:03:10.3320727Z Generating XML reports...
2023-01-11T22:03:10.3321074Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215240.xml
2023-01-11T22:03:10.3321353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3321466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3321750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3321887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3322111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiy61zv8h
2023-01-11T22:03:10.3322337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiy61zv8h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3322599Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3322605Z 
2023-01-11T22:03:10.3322678Z Running tests...
2023-01-11T22:03:10.3322878Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3323099Z   test_rref_forward_chain (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4410
2023-01-11T22:03:10.3323242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4411
2023-01-11T22:03:10.3323397Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4412
2023-01-11T22:03:10.3323551Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4413
2023-01-11T22:03:10.3323828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3323956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3324241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3324376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3324563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1bof9t8k
2023-01-11T22:03:10.3324746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1bof9t8k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3325019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3325143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3325429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3325568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3325754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwse84w99
2023-01-11T22:03:10.3325951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwse84w99/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3326224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3326347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3326617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3326749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3326932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6o3b4ljh
2023-01-11T22:03:10.3327127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6o3b4ljh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3327403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3327527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3327810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3327945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3328117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwo906xs8
2023-01-11T22:03:10.3328312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwo906xs8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3328480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3328674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3328864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3329027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3329095Z ok (2.844s)
2023-01-11T22:03:10.3329102Z 
2023-01-11T22:03:10.3329299Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3329374Z Ran 1 test in 2.844s
2023-01-11T22:03:10.3329380Z 
2023-01-11T22:03:10.3329430Z OK
2023-01-11T22:03:10.3329434Z 
2023-01-11T22:03:10.3329518Z Generating XML reports...
2023-01-11T22:03:10.3329861Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215245.xml
2023-01-11T22:03:10.3330136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3330264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3330546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3330681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3330867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9b9bi7tz
2023-01-11T22:03:10.3331048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9b9bi7tz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3331303Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3331309Z 
2023-01-11T22:03:10.3331383Z Running tests...
2023-01-11T22:03:10.3331578Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3331797Z   test_rref_get_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4633
2023-01-11T22:03:10.3331959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4634
2023-01-11T22:03:10.3332115Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4635
2023-01-11T22:03:10.3332266Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4636
2023-01-11T22:03:10.3332540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3332650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3332932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3333068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3333255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbxp8mm5
2023-01-11T22:03:10.3333452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbxp8mm5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3333728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3333852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3334134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3334268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3334439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo46b_3cx
2023-01-11T22:03:10.3334632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo46b_3cx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3334904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3335058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3335342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3335504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3335690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xbxljrm
2023-01-11T22:03:10.3335885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xbxljrm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3336146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3336270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3336551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3336686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3336874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpclpu88ug
2023-01-11T22:03:10.3337074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpclpu88ug/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3337238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3337403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3337566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3337716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3337783Z ok (2.742s)
2023-01-11T22:03:10.3337789Z 
2023-01-11T22:03:10.3337987Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3338062Z Ran 1 test in 2.743s
2023-01-11T22:03:10.3338068Z 
2023-01-11T22:03:10.3338133Z OK
2023-01-11T22:03:10.3338139Z 
2023-01-11T22:03:10.3338222Z Generating XML reports...
2023-01-11T22:03:10.3338564Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215250.xml
2023-01-11T22:03:10.3338838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3338949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3339228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3339363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3339548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80w8xta5
2023-01-11T22:03:10.3339740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80w8xta5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3340001Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3340007Z 
2023-01-11T22:03:10.3340080Z Running tests...
2023-01-11T22:03:10.3340282Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3340494Z   test_rref_leak (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4856
2023-01-11T22:03:10.3340638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4857
2023-01-11T22:03:10.3340791Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4858
2023-01-11T22:03:10.3340944Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4859
2023-01-11T22:03:10.3341222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3341346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3341785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3341972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3342161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2foo7c08
2023-01-11T22:03:10.3342343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2foo7c08/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3342620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3342746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3343030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3343165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3343352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8_wts5w
2023-01-11T22:03:10.3343550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8_wts5w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3343828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3343952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3344219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3344355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3344539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzi244wfv
2023-01-11T22:03:10.3344733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzi244wfv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3345007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3345134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3345419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3345555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3345728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwknm2pzr
2023-01-11T22:03:10.3345924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwknm2pzr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3346088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3346253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3346416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3346581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3346761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.3346941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.3347117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.3347278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.3347581Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3347879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3348172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3348504Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3349091Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2023-01-11T22:03:10.3349286Z Leaking RRef GloballyUniqueId(created_on=1, local_id=0) with fork Id GloballyUniqueId(created_on=1, local_id=1)
2023-01-11T22:03:10.3349292Z 
2023-01-11T22:03:10.3349835Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2023-01-11T22:03:10.3350023Z Leaking RRef GloballyUniqueId(created_on=0, local_id=0) with fork Id GloballyUniqueId(created_on=0, local_id=1)
2023-01-11T22:03:10.3350029Z 
2023-01-11T22:03:10.3350560Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2023-01-11T22:03:10.3350745Z Leaking RRef GloballyUniqueId(created_on=2, local_id=0) with fork Id GloballyUniqueId(created_on=2, local_id=1)
2023-01-11T22:03:10.3350753Z 
2023-01-11T22:03:10.3351281Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2023-01-11T22:03:10.3351461Z Leaking RRef GloballyUniqueId(created_on=3, local_id=0) with fork Id GloballyUniqueId(created_on=3, local_id=1)
2023-01-11T22:03:10.3351466Z 
2023-01-11T22:03:10.3351534Z ok (2.750s)
2023-01-11T22:03:10.3351540Z 
2023-01-11T22:03:10.3351743Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3351822Z Ran 1 test in 2.750s
2023-01-11T22:03:10.3351827Z 
2023-01-11T22:03:10.3351888Z OK
2023-01-11T22:03:10.3351894Z 
2023-01-11T22:03:10.3351967Z Generating XML reports...
2023-01-11T22:03:10.3352311Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215255.xml
2023-01-11T22:03:10.3352590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3352716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3352999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3353135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3353322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_wgykij7
2023-01-11T22:03:10.3353551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_wgykij7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3353837Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3353844Z 
2023-01-11T22:03:10.3353905Z Running tests...
2023-01-11T22:03:10.3354108Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3354327Z   test_rref_proxy_class (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5091
2023-01-11T22:03:10.3354485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5092
2023-01-11T22:03:10.3354640Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5093
2023-01-11T22:03:10.3354793Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5094
2023-01-11T22:03:10.3355068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3355195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3355467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3355604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3355790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsiv9ho1
2023-01-11T22:03:10.3355986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsiv9ho1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3356260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3356384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3356666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3356805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3356993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj64o2xd4
2023-01-11T22:03:10.3357174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj64o2xd4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3357445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3357569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3357852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3357986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3358169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpagn542t5
2023-01-11T22:03:10.3358363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpagn542t5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3358637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3358762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3359114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3359251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3359436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3zszr44u
2023-01-11T22:03:10.3359630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3zszr44u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3359796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3359961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3360161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3360361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3360417Z ok (2.744s)
2023-01-11T22:03:10.3360423Z 
2023-01-11T22:03:10.3360625Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3360702Z Ran 1 test in 2.744s
2023-01-11T22:03:10.3360708Z 
2023-01-11T22:03:10.3360772Z OK
2023-01-11T22:03:10.3360777Z 
2023-01-11T22:03:10.3360861Z Generating XML reports...
2023-01-11T22:03:10.3361206Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215259.xml
2023-01-11T22:03:10.3361482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3361607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3361891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3362014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3362200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89nvv8kp
2023-01-11T22:03:10.3362394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89nvv8kp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3362649Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3362655Z 
2023-01-11T22:03:10.3362729Z Running tests...
2023-01-11T22:03:10.3362927Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3363152Z   test_rref_proxy_class_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5314
2023-01-11T22:03:10.3363309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5315
2023-01-11T22:03:10.3363453Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5316
2023-01-11T22:03:10.3363611Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5317
2023-01-11T22:03:10.3363884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3364009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3364289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3364425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3364610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0lxtpdf
2023-01-11T22:03:10.3364805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0lxtpdf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3365082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3365195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3365477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3365611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3365796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4cdxh4px
2023-01-11T22:03:10.3365993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4cdxh4px/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3366262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3366386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3366665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3366826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3367042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4borq7bh
2023-01-11T22:03:10.3367238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4borq7bh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3367509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3367631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3367909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3368044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3368227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7s2ua5r3
2023-01-11T22:03:10.3368422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7s2ua5r3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3368575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3368740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3368903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3369066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3369135Z ok (2.738s)
2023-01-11T22:03:10.3369141Z 
2023-01-11T22:03:10.3369340Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3369417Z Ran 1 test in 2.738s
2023-01-11T22:03:10.3369422Z 
2023-01-11T22:03:10.3369485Z OK
2023-01-11T22:03:10.3369490Z 
2023-01-11T22:03:10.3369561Z Generating XML reports...
2023-01-11T22:03:10.3369905Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215304.xml
2023-01-11T22:03:10.3370182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3370307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3370588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3370724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3370907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnwo687u
2023-01-11T22:03:10.3371100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnwo687u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3371353Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3371361Z 
2023-01-11T22:03:10.3371423Z Running tests...
2023-01-11T22:03:10.3371619Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3371846Z   test_rref_proxy_non_exist (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5537
2023-01-11T22:03:10.3372002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5538
2023-01-11T22:03:10.3372156Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5539
2023-01-11T22:03:10.3372309Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5540
2023-01-11T22:03:10.3372582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3372706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3372973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3373141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3373358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3peq2vtj
2023-01-11T22:03:10.3373553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3peq2vtj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3373827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3373951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3374230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3374364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3374548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqf6rlouk
2023-01-11T22:03:10.3374730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqf6rlouk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3375006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3375129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3375410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3375544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3375730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbyxtz_w
2023-01-11T22:03:10.3375925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbyxtz_w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3376197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3376320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3376589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3376724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3376906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplg6h3yto
2023-01-11T22:03:10.3377101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplg6h3yto/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3377266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3377431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3377614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3377849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3377936Z ok (2.743s)
2023-01-11T22:03:10.3377944Z 
2023-01-11T22:03:10.3378152Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3378230Z Ran 1 test in 2.743s
2023-01-11T22:03:10.3378238Z 
2023-01-11T22:03:10.3378299Z OK
2023-01-11T22:03:10.3378304Z 
2023-01-11T22:03:10.3378389Z Generating XML reports...
2023-01-11T22:03:10.3378727Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215309.xml
2023-01-11T22:03:10.3378998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3379122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3379404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3379527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3379759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiix5ebem
2023-01-11T22:03:10.3379981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiix5ebem/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3380241Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3380246Z 
2023-01-11T22:03:10.3380320Z Running tests...
2023-01-11T22:03:10.3380516Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3380737Z   test_rref_proxy_reuse (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5760
2023-01-11T22:03:10.3380892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5761
2023-01-11T22:03:10.3381035Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5762
2023-01-11T22:03:10.3381187Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5763
2023-01-11T22:03:10.3381466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3381594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3381973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3382110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3382300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpus0afbsr
2023-01-11T22:03:10.3382501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpus0afbsr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3382778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3382889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3383172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3383310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3383499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkq7s873e
2023-01-11T22:03:10.3383695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkq7s873e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3383968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3384094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3384376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3384501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3384689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpncrirwee
2023-01-11T22:03:10.3384889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpncrirwee/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3385164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3385288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3385569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3385704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3385890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxlt1mni
2023-01-11T22:03:10.3386085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxlt1mni/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3386238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3386459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3386658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3386822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3386892Z ok (2.750s)
2023-01-11T22:03:10.3386897Z 
2023-01-11T22:03:10.3387100Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3387178Z Ran 1 test in 2.751s
2023-01-11T22:03:10.3387184Z 
2023-01-11T22:03:10.3387246Z OK
2023-01-11T22:03:10.3387251Z 
2023-01-11T22:03:10.3387324Z Generating XML reports...
2023-01-11T22:03:10.3387671Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215313.xml
2023-01-11T22:03:10.3387946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3388074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3388374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3388580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3388844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn45tq4ze
2023-01-11T22:03:10.3389047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn45tq4ze/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3389306Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3389312Z 
2023-01-11T22:03:10.3389373Z Running tests...
2023-01-11T22:03:10.3389571Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3389790Z   test_rref_proxy_tensor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5983
2023-01-11T22:03:10.3389950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5984
2023-01-11T22:03:10.3390107Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5985
2023-01-11T22:03:10.3390260Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5986
2023-01-11T22:03:10.3390536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3390660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3390930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3391065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3391251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fmj93fc
2023-01-11T22:03:10.3391447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fmj93fc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3391722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3391846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3392128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3392262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3392445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkicb9vtd
2023-01-11T22:03:10.3392628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkicb9vtd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3392903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3393073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3393352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3393518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3393705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7bkwkuh
2023-01-11T22:03:10.3393900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7bkwkuh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3394172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3394294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3394560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3394693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3394880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplng8fip8
2023-01-11T22:03:10.3395076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplng8fip8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3395239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3395405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3395567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3395728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3395783Z ok (2.749s)
2023-01-11T22:03:10.3395789Z 
2023-01-11T22:03:10.3395989Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3396066Z Ran 1 test in 2.749s
2023-01-11T22:03:10.3396071Z 
2023-01-11T22:03:10.3396135Z OK
2023-01-11T22:03:10.3396140Z 
2023-01-11T22:03:10.3396227Z Generating XML reports...
2023-01-11T22:03:10.3396570Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215318.xml
2023-01-11T22:03:10.3396846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3396969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3397251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3397373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3397556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptne1d5vq
2023-01-11T22:03:10.3397751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptne1d5vq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3398009Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3398014Z 
2023-01-11T22:03:10.3398087Z Running tests...
2023-01-11T22:03:10.3398285Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3398509Z   test_rref_proxy_tensor_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6206
2023-01-11T22:03:10.3398666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6207
2023-01-11T22:03:10.3398806Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6208
2023-01-11T22:03:10.3399025Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6209
2023-01-11T22:03:10.3399304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3399430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3399765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3399996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3400269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8jxm3y4
2023-01-11T22:03:10.3400532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8jxm3y4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3400936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3401102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3401493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3401665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3401857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhzcy9ef
2023-01-11T22:03:10.3402059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhzcy9ef/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3402412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3402587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3402930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3403055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3403242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08kca6x7
2023-01-11T22:03:10.3403438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08kca6x7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3403714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3403842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3404129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3404309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3404508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy07z2zu8
2023-01-11T22:03:10.3404768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy07z2zu8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3404978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3405145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3405305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3405471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3405538Z ok (2.648s)
2023-01-11T22:03:10.3405546Z 
2023-01-11T22:03:10.3405752Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3405829Z Ran 1 test in 2.649s
2023-01-11T22:03:10.3405834Z 
2023-01-11T22:03:10.3405897Z OK
2023-01-11T22:03:10.3405902Z 
2023-01-11T22:03:10.3405974Z Generating XML reports...
2023-01-11T22:03:10.3406318Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215323.xml
2023-01-11T22:03:10.3406594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3406719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3407002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3407199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3407386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44zbl3vj
2023-01-11T22:03:10.3407633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44zbl3vj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3407892Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3407898Z 
2023-01-11T22:03:10.3407959Z Running tests...
2023-01-11T22:03:10.3408156Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3408392Z   test_rref_py_pickle_not_supported (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6429
2023-01-11T22:03:10.3408551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6430
2023-01-11T22:03:10.3408705Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6431
2023-01-11T22:03:10.3408860Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6432
2023-01-11T22:03:10.3409136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3409262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3409530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3409666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3409852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzccfqri0
2023-01-11T22:03:10.3410051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzccfqri0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3410327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3410454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3410739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3410876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3411062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpykr6giyi
2023-01-11T22:03:10.3411246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpykr6giyi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3411521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3411647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3411927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3412063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3412251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_l7300qq
2023-01-11T22:03:10.3412445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_l7300qq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3412716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3412841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3413108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3413242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3413425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5k_rz6p
2023-01-11T22:03:10.3413619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5k_rz6p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3413815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3414033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3414198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3414361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3414416Z ok (2.745s)
2023-01-11T22:03:10.3414421Z 
2023-01-11T22:03:10.3414619Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3414695Z Ran 1 test in 2.746s
2023-01-11T22:03:10.3414701Z 
2023-01-11T22:03:10.3414764Z OK
2023-01-11T22:03:10.3414769Z 
2023-01-11T22:03:10.3414856Z Generating XML reports...
2023-01-11T22:03:10.3415200Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215327.xml
2023-01-11T22:03:10.3415483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3415612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3415894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3416017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3416205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd77vg4u2
2023-01-11T22:03:10.3416402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd77vg4u2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3416660Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3416666Z 
2023-01-11T22:03:10.3416741Z Running tests...
2023-01-11T22:03:10.3416943Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3417161Z   test_rref_str (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6652
2023-01-11T22:03:10.3417319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6653
2023-01-11T22:03:10.3417460Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6654
2023-01-11T22:03:10.3417613Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6655
2023-01-11T22:03:10.3417886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3418014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3418294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3418430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3418615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp638ra3hd
2023-01-11T22:03:10.3418812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp638ra3hd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3419088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3419199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3419480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3419615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3419801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd0y_7dp1
2023-01-11T22:03:10.3419995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd0y_7dp1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3420267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3420423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3420731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3420854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3421038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp259py3xe
2023-01-11T22:03:10.3421231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp259py3xe/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3421504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3421629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3422025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3422163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3422351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppewva26h
2023-01-11T22:03:10.3422548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppewva26h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3422704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3422867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3423029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3423192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3423262Z ok (2.751s)
2023-01-11T22:03:10.3423269Z 
2023-01-11T22:03:10.3423469Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3423549Z Ran 1 test in 2.751s
2023-01-11T22:03:10.3423554Z 
2023-01-11T22:03:10.3423616Z OK
2023-01-11T22:03:10.3423622Z 
2023-01-11T22:03:10.3423693Z Generating XML reports...
2023-01-11T22:03:10.3424038Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215332.xml
2023-01-11T22:03:10.3424313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3424439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3424722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3424859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3425045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbe636iap
2023-01-11T22:03:10.3425239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbe636iap/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3425499Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3425507Z 
2023-01-11T22:03:10.3425568Z Running tests...
2023-01-11T22:03:10.3425767Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3425982Z   test_rref_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6875
2023-01-11T22:03:10.3426139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6876
2023-01-11T22:03:10.3426294Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6877
2023-01-11T22:03:10.3426448Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6878
2023-01-11T22:03:10.3426725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3426909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3427216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3427356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3427539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1_yvg81
2023-01-11T22:03:10.3427732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1_yvg81/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3428010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3428135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3428415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3428553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3428741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7cn4aag
2023-01-11T22:03:10.3428925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7cn4aag/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3429198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3429323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3429601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3429736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3429921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplghnxu2l
2023-01-11T22:03:10.3430117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplghnxu2l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3430395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3430509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3430789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3430927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3431110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1d2awa0
2023-01-11T22:03:10.3431305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1d2awa0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3431472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3431636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3431802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3431965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3432023Z ok (4.647s)
2023-01-11T22:03:10.3432029Z 
2023-01-11T22:03:10.3432227Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3432303Z Ran 1 test in 4.647s
2023-01-11T22:03:10.3432310Z 
2023-01-11T22:03:10.3432372Z OK
2023-01-11T22:03:10.3432377Z 
2023-01-11T22:03:10.3432462Z Generating XML reports...
2023-01-11T22:03:10.3432799Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215337.xml
2023-01-11T22:03:10.3433073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3433198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3433466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3433639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3433849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lz6uady
2023-01-11T22:03:10.3434045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lz6uady/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3434303Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3434309Z 
2023-01-11T22:03:10.3434382Z Running tests...
2023-01-11T22:03:10.3434580Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3434803Z   test_rref_type_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7098
2023-01-11T22:03:10.3434962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7099
2023-01-11T22:03:10.3435106Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7100
2023-01-11T22:03:10.3435261Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7101
2023-01-11T22:03:10.3435535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3435661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3435943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3436080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3436265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdrc5lti
2023-01-11T22:03:10.3436461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdrc5lti/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3436733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3436847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3437129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3437264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3437450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfub57oxm
2023-01-11T22:03:10.3437646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfub57oxm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3437917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3438041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3438322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3438449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3438633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1niu4_1s
2023-01-11T22:03:10.3438886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1niu4_1s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3439168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3439293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3439574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3439710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3439896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2_ky43p
2023-01-11T22:03:10.3440128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2_ky43p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3440306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3440474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3440638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3440803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3440873Z ok (2.638s)
2023-01-11T22:03:10.3440879Z 
2023-01-11T22:03:10.3441079Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3441156Z Ran 1 test in 2.638s
2023-01-11T22:03:10.3441162Z 
2023-01-11T22:03:10.3441223Z OK
2023-01-11T22:03:10.3441229Z 
2023-01-11T22:03:10.3441300Z Generating XML reports...
2023-01-11T22:03:10.3441639Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215343.xml
2023-01-11T22:03:10.3441915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3442041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3442324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3442459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3442646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg5mpv8um
2023-01-11T22:03:10.3442841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg5mpv8um/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3443099Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3443104Z 
2023-01-11T22:03:10.3443165Z Running tests...
2023-01-11T22:03:10.3443365Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3443591Z   test_rref_type_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7321
2023-01-11T22:03:10.3443747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7322
2023-01-11T22:03:10.3443901Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7323
2023-01-11T22:03:10.3444055Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7324
2023-01-11T22:03:10.3444331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3444458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3444730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3444871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3445056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmty7whd6
2023-01-11T22:03:10.3445261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmty7whd6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3445535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3445659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3445943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3446079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3446265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9o104zl
2023-01-11T22:03:10.3446447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9o104zl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3446762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3446915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3447199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3447334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3447517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpss13cjd7
2023-01-11T22:03:10.3447713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpss13cjd7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3447986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3448098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3448382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3448520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3448703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01u16evi
2023-01-11T22:03:10.3448896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01u16evi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3449064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3449229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3449392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3449554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3449609Z ok (2.644s)
2023-01-11T22:03:10.3449616Z 
2023-01-11T22:03:10.3449815Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3449890Z Ran 1 test in 2.644s
2023-01-11T22:03:10.3449895Z 
2023-01-11T22:03:10.3449959Z OK
2023-01-11T22:03:10.3449965Z 
2023-01-11T22:03:10.3450050Z Generating XML reports...
2023-01-11T22:03:10.3450387Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215348.xml
2023-01-11T22:03:10.3450665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3450789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3451057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3451192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3451374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_sa6kjot
2023-01-11T22:03:10.3451570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_sa6kjot/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3451827Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3451833Z 
2023-01-11T22:03:10.3451907Z Running tests...
2023-01-11T22:03:10.3452104Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3452331Z   test_rref_type_owner_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7544
2023-01-11T22:03:10.3452487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7545
2023-01-11T22:03:10.3452626Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7546
2023-01-11T22:03:10.3452779Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7547
2023-01-11T22:03:10.3453055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3453214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3453537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3453677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3453863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0t0t2oe
2023-01-11T22:03:10.3454057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0t0t2oe/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3454331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3454441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3454722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3454858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3455046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmj7be7sb
2023-01-11T22:03:10.3455239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmj7be7sb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3455512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3455635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3455914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3456035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3456218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tpin1rc
2023-01-11T22:03:10.3456490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3456616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3456897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3457031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3457217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63bpqq2w
2023-01-11T22:03:10.3457413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tpin1rc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3457603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63bpqq2w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3457756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3457923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3458085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3458247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3458314Z ok (2.756s)
2023-01-11T22:03:10.3458321Z 
2023-01-11T22:03:10.3458520Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3458597Z Ran 1 test in 2.756s
2023-01-11T22:03:10.3458603Z 
2023-01-11T22:03:10.3458663Z OK
2023-01-11T22:03:10.3458668Z 
2023-01-11T22:03:10.3458739Z Generating XML reports...
2023-01-11T22:03:10.3459078Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215352.xml
2023-01-11T22:03:10.3459355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3459510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3459789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3459950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3460136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3qwh5ita
2023-01-11T22:03:10.3460332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3qwh5ita/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3460586Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3460592Z 
2023-01-11T22:03:10.3460652Z Running tests...
2023-01-11T22:03:10.3460849Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3461076Z   test_rref_type_owner_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7767
2023-01-11T22:03:10.3461234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7768
2023-01-11T22:03:10.3461387Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7769
2023-01-11T22:03:10.3461541Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7770
2023-01-11T22:03:10.3461918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3462044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3462311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3462449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3462635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81yl9pan
2023-01-11T22:03:10.3462829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81yl9pan/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3463105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3463232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3463514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3463650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3463833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6dbf_919
2023-01-11T22:03:10.3464012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6dbf_919/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3464282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3464406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3464687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3464824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3465008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpws8uyhly
2023-01-11T22:03:10.3465205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpws8uyhly/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3465477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3465587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3465865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3465998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3466228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1q5j7eq
2023-01-11T22:03:10.3466485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1q5j7eq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3466653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3466818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3466982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3467146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3467201Z ok (2.752s)
2023-01-11T22:03:10.3467207Z 
2023-01-11T22:03:10.3467408Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3467484Z Ran 1 test in 2.752s
2023-01-11T22:03:10.3467489Z 
2023-01-11T22:03:10.3467551Z OK
2023-01-11T22:03:10.3467558Z 
2023-01-11T22:03:10.3467643Z Generating XML reports...
2023-01-11T22:03:10.3467988Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215357.xml
2023-01-11T22:03:10.3468265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3468390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3468657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3468793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3468975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7f7j2ijk
2023-01-11T22:03:10.3469169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7f7j2ijk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3469426Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3469435Z 
2023-01-11T22:03:10.3469510Z Running tests...
2023-01-11T22:03:10.3469709Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3469933Z   test_rref_type_slow_init (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7990
2023-01-11T22:03:10.3470091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7991
2023-01-11T22:03:10.3470230Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7992
2023-01-11T22:03:10.3470384Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7993
2023-01-11T22:03:10.3470655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3470779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3471145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3471313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3471503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc889f4ew
2023-01-11T22:03:10.3471742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc889f4ew/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3472137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3472249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3472533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3472667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3472850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqh_1q03f
2023-01-11T22:03:10.3473101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqh_1q03f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3473407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3473532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3473813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3473935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3474122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdb9btpj
2023-01-11T22:03:10.3474322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdb9btpj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3474593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3474723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3475005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3475140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3475326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9cupx2c
2023-01-11T22:03:10.3475519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9cupx2c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3475672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3475837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3475999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3476163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3476233Z ok (3.660s)
2023-01-11T22:03:10.3476240Z 
2023-01-11T22:03:10.3476438Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3476513Z Ran 1 test in 3.660s
2023-01-11T22:03:10.3476519Z 
2023-01-11T22:03:10.3476579Z OK
2023-01-11T22:03:10.3476584Z 
2023-01-11T22:03:10.3476656Z Generating XML reports...
2023-01-11T22:03:10.3476992Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215402.xml
2023-01-11T22:03:10.3477265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3477391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3477746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3477882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3478071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpktyngb8o
2023-01-11T22:03:10.3478271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpktyngb8o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3478525Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3478531Z 
2023-01-11T22:03:10.3478591Z Running tests...
2023-01-11T22:03:10.3478787Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3479090Z   test_rref_type_with_error_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8213
2023-01-11T22:03:10.3479250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8214
2023-01-11T22:03:10.3479476Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8215
2023-01-11T22:03:10.3479697Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8216
2023-01-11T22:03:10.3480013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3480140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3480411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3480548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3480733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuv0chben
2023-01-11T22:03:10.3481008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3481135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3481418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3481557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3481744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbg9glkj
2023-01-11T22:03:10.3481938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuv0chben/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3482115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbg9glkj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3482388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3482511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3482791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3482926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3483113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7p050t9m
2023-01-11T22:03:10.3483309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7p050t9m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3483581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3483691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3483972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3484107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3484289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpra_i_i80
2023-01-11T22:03:10.3484480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpra_i_i80/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3484647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3484813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3484978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3485140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3485219Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3485397Z ValueError('Expected error')
2023-01-11T22:03:10.3485500Z Traceback (most recent call last):
2023-01-11T22:03:10.3485768Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3485905Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3486189Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3486319Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3486393Z ValueError: Expected error
2023-01-11T22:03:10.3486411Z 
2023-01-11T22:03:10.3486489Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3487387Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2023-01-11T22:03:10.3487484Z Traceback (most recent call last):
2023-01-11T22:03:10.3487750Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3487886Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3488156Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2023-01-11T22:03:10.3488257Z     rref_type = type(rref.local_value())
2023-01-11T22:03:10.3488528Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 234, in _handle_exception
2023-01-11T22:03:10.3488597Z     raise exc
2023-01-11T22:03:10.3488694Z ValueError: On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3488816Z ValueError('Expected error')
2023-01-11T22:03:10.3488910Z Traceback (most recent call last):
2023-01-11T22:03:10.3489169Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3489303Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3489587Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3489680Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3489765Z ValueError: Expected error
2023-01-11T22:03:10.3489771Z 
2023-01-11T22:03:10.3489776Z 
2023-01-11T22:03:10.3489854Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3489977Z ValueError('Expected error')
2023-01-11T22:03:10.3490116Z Traceback (most recent call last):
2023-01-11T22:03:10.3490521Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3490667Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3490951Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3491042Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3491127Z ValueError: Expected error
2023-01-11T22:03:10.3491132Z 
2023-01-11T22:03:10.3491209Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3492127Z ValueError('On WorkerInfo(id=2, name=worker2):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2023-01-11T22:03:10.3492228Z Traceback (most recent call last):
2023-01-11T22:03:10.3492489Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3492624Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3492891Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2023-01-11T22:03:10.3492990Z     rref_type = type(rref.local_value())
2023-01-11T22:03:10.3493255Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 234, in _handle_exception
2023-01-11T22:03:10.3493370Z     raise exc
2023-01-11T22:03:10.3493468Z ValueError: On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3493617Z ValueError('Expected error')
2023-01-11T22:03:10.3493713Z Traceback (most recent call last):
2023-01-11T22:03:10.3493967Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3494100Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3494382Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3494475Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3494560Z ValueError: Expected error
2023-01-11T22:03:10.3494565Z 
2023-01-11T22:03:10.3494570Z 
2023-01-11T22:03:10.3494646Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3494766Z ValueError('Expected error')
2023-01-11T22:03:10.3494860Z Traceback (most recent call last):
2023-01-11T22:03:10.3495116Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3495252Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3495534Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3495626Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3495712Z ValueError: Expected error
2023-01-11T22:03:10.3495717Z 
2023-01-11T22:03:10.3495793Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3496656Z ValueError('On WorkerInfo(id=0, name=worker0):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2023-01-11T22:03:10.3496752Z Traceback (most recent call last):
2023-01-11T22:03:10.3497017Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3497152Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3497418Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2023-01-11T22:03:10.3497516Z     rref_type = type(rref.local_value())
2023-01-11T22:03:10.3497784Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 234, in _handle_exception
2023-01-11T22:03:10.3497851Z     raise exc
2023-01-11T22:03:10.3497949Z ValueError: On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3498071Z ValueError('Expected error')
2023-01-11T22:03:10.3498163Z Traceback (most recent call last):
2023-01-11T22:03:10.3498446Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3498665Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3499169Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3499321Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3499465Z ValueError: Expected error
2023-01-11T22:03:10.3499473Z 
2023-01-11T22:03:10.3499480Z 
2023-01-11T22:03:10.3499629Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3499815Z ValueError('Expected error')
2023-01-11T22:03:10.3499965Z Traceback (most recent call last):
2023-01-11T22:03:10.3500421Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3500647Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3501190Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3501347Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3501542Z ValueError: Expected error
2023-01-11T22:03:10.3501552Z 
2023-01-11T22:03:10.3501817Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3502982Z ValueError('On WorkerInfo(id=3, name=worker3):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2023-01-11T22:03:10.3503079Z Traceback (most recent call last):
2023-01-11T22:03:10.3503344Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3503483Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3503753Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2023-01-11T22:03:10.3503854Z     rref_type = type(rref.local_value())
2023-01-11T22:03:10.3504123Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 234, in _handle_exception
2023-01-11T22:03:10.3504193Z     raise exc
2023-01-11T22:03:10.3504307Z ValueError: On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3504415Z ValueError('Expected error')
2023-01-11T22:03:10.3504509Z Traceback (most recent call last):
2023-01-11T22:03:10.3504768Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3504904Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3505186Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3505282Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3505368Z ValueError: Expected error
2023-01-11T22:03:10.3505374Z 
2023-01-11T22:03:10.3505379Z 
2023-01-11T22:03:10.3505446Z ok (2.741s)
2023-01-11T22:03:10.3505451Z 
2023-01-11T22:03:10.3505636Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3505712Z Ran 1 test in 2.741s
2023-01-11T22:03:10.3505717Z 
2023-01-11T22:03:10.3505778Z OK
2023-01-11T22:03:10.3505783Z 
2023-01-11T22:03:10.3505869Z Generating XML reports...
2023-01-11T22:03:10.3506213Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215407.xml
2023-01-11T22:03:10.3506490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3506616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3507017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3507241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3507560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmibzksrb
2023-01-11T22:03:10.3507894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmibzksrb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3508345Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3508353Z 
2023-01-11T22:03:10.3508470Z Running tests...
2023-01-11T22:03:10.3508807Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3509214Z   test_rref_type_with_error_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8436
2023-01-11T22:03:10.3509568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8437
2023-01-11T22:03:10.3509837Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8438
2023-01-11T22:03:10.3510146Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8439
2023-01-11T22:03:10.3510645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3510858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3511215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3511356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3511545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3hn6w5x
2023-01-11T22:03:10.3511821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3511950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3512234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3512358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3512547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfhil2oa
2023-01-11T22:03:10.3512744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3hn6w5x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3512939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfhil2oa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3513217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3513342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3513627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3513766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3513937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsuuo40fe
2023-01-11T22:03:10.3514133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsuuo40fe/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3514404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3514528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3514809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3514943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3515129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpma88l80o
2023-01-11T22:03:10.3515325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpma88l80o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3515491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3515644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3515806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3515966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3516059Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3516181Z ValueError('Expected error')
2023-01-11T22:03:10.3516275Z Traceback (most recent call last):
2023-01-11T22:03:10.3516538Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3516700Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3517016Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3517109Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3517197Z ValueError: Expected error
2023-01-11T22:03:10.3517202Z 
2023-01-11T22:03:10.3517295Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3518161Z ValueError('On WorkerInfo(id=0, name=worker0):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2023-01-11T22:03:10.3518256Z Traceback (most recent call last):
2023-01-11T22:03:10.3518519Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3518656Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3518987Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2023-01-11T22:03:10.3519075Z     rref_type = type(rref.local_value())
2023-01-11T22:03:10.3519342Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 234, in _handle_exception
2023-01-11T22:03:10.3519410Z     raise exc
2023-01-11T22:03:10.3519521Z ValueError: On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3519643Z ValueError('Expected error')
2023-01-11T22:03:10.3519736Z Traceback (most recent call last):
2023-01-11T22:03:10.3519994Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3520128Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3520401Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3520497Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3520584Z ValueError: Expected error
2023-01-11T22:03:10.3520589Z 
2023-01-11T22:03:10.3520594Z 
2023-01-11T22:03:10.3520685Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3520805Z ValueError('Expected error')
2023-01-11T22:03:10.3520898Z Traceback (most recent call last):
2023-01-11T22:03:10.3521156Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3521289Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3521555Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3521647Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3521734Z ValueError: Expected error
2023-01-11T22:03:10.3521739Z 
2023-01-11T22:03:10.3521828Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3522691Z ValueError('On WorkerInfo(id=2, name=worker2):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2023-01-11T22:03:10.3522786Z Traceback (most recent call last):
2023-01-11T22:03:10.3523046Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3523182Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3523449Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2023-01-11T22:03:10.3523600Z     rref_type = type(rref.local_value())
2023-01-11T22:03:10.3523898Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 234, in _handle_exception
2023-01-11T22:03:10.3523969Z     raise exc
2023-01-11T22:03:10.3524079Z ValueError: On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3524202Z ValueError('Expected error')
2023-01-11T22:03:10.3524295Z Traceback (most recent call last):
2023-01-11T22:03:10.3524554Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3524690Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3524991Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3525132Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3525271Z ValueError: Expected error
2023-01-11T22:03:10.3525282Z 
2023-01-11T22:03:10.3525288Z 
2023-01-11T22:03:10.3525432Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3525609Z ValueError('Expected error')
2023-01-11T22:03:10.3525703Z Traceback (most recent call last):
2023-01-11T22:03:10.3526010Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3526225Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3526514Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3526606Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3526691Z ValueError: Expected error
2023-01-11T22:03:10.3526698Z 
2023-01-11T22:03:10.3526788Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3527655Z ValueError('On WorkerInfo(id=3, name=worker3):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2023-01-11T22:03:10.3527752Z Traceback (most recent call last):
2023-01-11T22:03:10.3528015Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3528149Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3528415Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2023-01-11T22:03:10.3528514Z     rref_type = type(rref.local_value())
2023-01-11T22:03:10.3528768Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 234, in _handle_exception
2023-01-11T22:03:10.3528838Z     raise exc
2023-01-11T22:03:10.3528946Z ValueError: On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3529068Z ValueError('Expected error')
2023-01-11T22:03:10.3529162Z Traceback (most recent call last):
2023-01-11T22:03:10.3529419Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3529551Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3529820Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3529912Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3529998Z ValueError: Expected error
2023-01-11T22:03:10.3530003Z 
2023-01-11T22:03:10.3530008Z 
2023-01-11T22:03:10.3530098Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3530272Z ValueError('Expected error')
2023-01-11T22:03:10.3530431Z Traceback (most recent call last):
2023-01-11T22:03:10.3530962Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3531117Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3531424Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3531519Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3531605Z ValueError: Expected error
2023-01-11T22:03:10.3531610Z 
2023-01-11T22:03:10.3531701Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3532571Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2023-01-11T22:03:10.3532668Z Traceback (most recent call last):
2023-01-11T22:03:10.3532936Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3533070Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3533335Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2023-01-11T22:03:10.3533434Z     rref_type = type(rref.local_value())
2023-01-11T22:03:10.3533689Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 234, in _handle_exception
2023-01-11T22:03:10.3533756Z     raise exc
2023-01-11T22:03:10.3533866Z ValueError: On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3533987Z ValueError('Expected error')
2023-01-11T22:03:10.3534080Z Traceback (most recent call last):
2023-01-11T22:03:10.3534338Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3534474Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3534748Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3534841Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3534927Z ValueError: Expected error
2023-01-11T22:03:10.3534932Z 
2023-01-11T22:03:10.3534937Z 
2023-01-11T22:03:10.3535005Z ok (2.645s)
2023-01-11T22:03:10.3535011Z 
2023-01-11T22:03:10.3535212Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3535287Z Ran 1 test in 2.646s
2023-01-11T22:03:10.3535292Z 
2023-01-11T22:03:10.3535354Z OK
2023-01-11T22:03:10.3535359Z 
2023-01-11T22:03:10.3535443Z Generating XML reports...
2023-01-11T22:03:10.3535784Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215412.xml
2023-01-11T22:03:10.3536050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3536176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3536458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3536597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3536785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpis0y410x
2023-01-11T22:03:10.3536983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpis0y410x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3537237Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3537243Z 
2023-01-11T22:03:10.3537315Z Running tests...
2023-01-11T22:03:10.3537514Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3537744Z   test_scalar_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8659
2023-01-11T22:03:10.3537929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8660
2023-01-11T22:03:10.3538084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8661
2023-01-11T22:03:10.3538237Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8662
2023-01-11T22:03:10.3538537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3538736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3539120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3539260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3539440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0e7ob04
2023-01-11T22:03:10.3539639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0e7ob04/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3539914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3540040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3540325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3540462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3540710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkldgaewa
2023-01-11T22:03:10.3541048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkldgaewa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3541409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3541524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3541931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3542069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3542254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_57tzzr7
2023-01-11T22:03:10.3542452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_57tzzr7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3542726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3542851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3543130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3543256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3543442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3o6us8e7
2023-01-11T22:03:10.3543635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3o6us8e7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3543804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3543970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3544135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3544299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3544368Z ok (2.749s)
2023-01-11T22:03:10.3544376Z 
2023-01-11T22:03:10.3544578Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3544712Z Ran 1 test in 2.750s
2023-01-11T22:03:10.3544718Z 
2023-01-11T22:03:10.3544781Z OK
2023-01-11T22:03:10.3544786Z 
2023-01-11T22:03:10.3544871Z Generating XML reports...
2023-01-11T22:03:10.3545254Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215416.xml
2023-01-11T22:03:10.3545534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3545659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3545944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3546080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3546253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppu8id3dn
2023-01-11T22:03:10.3546516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppu8id3dn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3546794Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3546799Z 
2023-01-11T22:03:10.3546873Z Running tests...
2023-01-11T22:03:10.3547072Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3547284Z   test_self_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8882
2023-01-11T22:03:10.3547441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8883
2023-01-11T22:03:10.3547595Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8884
2023-01-11T22:03:10.3547748Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8885
2023-01-11T22:03:10.3548011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3548139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3548591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3548739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3548924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2boudbvu
2023-01-11T22:03:10.3549120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2boudbvu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3549398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3549521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3549804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3549978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3550167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnm_lmukz
2023-01-11T22:03:10.3550363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnm_lmukz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3550639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3550764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3551047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3551181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3551364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95ucea2s
2023-01-11T22:03:10.3551544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95ucea2s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3551861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3552026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3552310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3552445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3552632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ukavr21
2023-01-11T22:03:10.3552824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ukavr21/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3552989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3553170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3553369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3553534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3553604Z ok (2.749s)
2023-01-11T22:03:10.3553610Z 
2023-01-11T22:03:10.3553812Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3553888Z Ran 1 test in 2.749s
2023-01-11T22:03:10.3553894Z 
2023-01-11T22:03:10.3553953Z OK
2023-01-11T22:03:10.3553959Z 
2023-01-11T22:03:10.3554042Z Generating XML reports...
2023-01-11T22:03:10.3554383Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215421.xml
2023-01-11T22:03:10.3554646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3554771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3555055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3555195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3555381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8t17ta5c
2023-01-11T22:03:10.3555575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8t17ta5c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3555831Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3555837Z 
2023-01-11T22:03:10.3555910Z Running tests...
2023-01-11T22:03:10.3556094Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3556315Z   test_self_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9105
2023-01-11T22:03:10.3556472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9106
2023-01-11T22:03:10.3556629Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9107
2023-01-11T22:03:10.3556784Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9108
2023-01-11T22:03:10.3557058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3557184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3557465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3557600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3557773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1tp2it0e
2023-01-11T22:03:10.3557968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1tp2it0e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3558241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3558404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3558715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3558919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3559107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ls4q0ai
2023-01-11T22:03:10.3559382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3559505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3559773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3559908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3560096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0bha91mg
2023-01-11T22:03:10.3560291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ls4q0ai/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3560482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0bha91mg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3560758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3560882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3561160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3561283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3561467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzsvyahu3
2023-01-11T22:03:10.3561665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzsvyahu3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3561832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3562001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3562164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3562327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3562396Z ok (2.742s)
2023-01-11T22:03:10.3562402Z 
2023-01-11T22:03:10.3562600Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3562663Z Ran 1 test in 2.742s
2023-01-11T22:03:10.3562671Z 
2023-01-11T22:03:10.3562759Z OK
2023-01-11T22:03:10.3562767Z 
2023-01-11T22:03:10.3562901Z Generating XML reports...
2023-01-11T22:03:10.3563494Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215426.xml
2023-01-11T22:03:10.3564002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3564219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3564719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3564958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3565269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxc_ldb0n
2023-01-11T22:03:10.3565604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxc_ldb0n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3565875Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3565881Z 
2023-01-11T22:03:10.3566000Z Running tests...
2023-01-11T22:03:10.3566200Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3566557Z   test_self_remote_rref_as_remote_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9328
2023-01-11T22:03:10.3566844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9329
2023-01-11T22:03:10.3567116Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9330
2023-01-11T22:03:10.3567377Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9331
2023-01-11T22:03:10.3567864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3568086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3568547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3568744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3568933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5m9y9z1
2023-01-11T22:03:10.3569133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5m9y9z1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3569411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3569536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3569807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3569943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3570131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnr9047p
2023-01-11T22:03:10.3570325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnr9047p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3570599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3570726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3571008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3571143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3571326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4amgc41v
2023-01-11T22:03:10.3571509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4amgc41v/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3571781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3571904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3572186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3572326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3572632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxye6zc9d
2023-01-11T22:03:10.3572977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxye6zc9d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3573267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3573559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3573832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3574115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3574226Z ok (2.742s)
2023-01-11T22:03:10.3574297Z 
2023-01-11T22:03:10.3574642Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3574772Z Ran 1 test in 2.742s
2023-01-11T22:03:10.3574781Z 
2023-01-11T22:03:10.3574928Z OK
2023-01-11T22:03:10.3574936Z 
2023-01-11T22:03:10.3575083Z Generating XML reports...
2023-01-11T22:03:10.3575597Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215430.xml
2023-01-11T22:03:10.3575863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3575988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3576268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3576405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3576592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdvmbcag0
2023-01-11T22:03:10.3576792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdvmbcag0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3577048Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3577054Z 
2023-01-11T22:03:10.3577127Z Running tests...
2023-01-11T22:03:10.3577312Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3577546Z   test_self_remote_rref_as_rpc_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9551
2023-01-11T22:03:10.3577701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9552
2023-01-11T22:03:10.3577852Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9553
2023-01-11T22:03:10.3578006Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9554
2023-01-11T22:03:10.3578285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3578411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3578692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3578829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3579003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpapc5vmw1
2023-01-11T22:03:10.3579198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpapc5vmw1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3579474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3579599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3579881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3580020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3580210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4yrrxkr
2023-01-11T22:03:10.3580407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4yrrxkr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3580798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3580998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3581486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3581795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3582053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_in37iag
2023-01-11T22:03:10.3582332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_in37iag/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3582659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3582784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3583067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3583189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3583374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwprwoqk
2023-01-11T22:03:10.3583571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwprwoqk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3583738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3583906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3584073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3584235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3584304Z ok (2.841s)
2023-01-11T22:03:10.3584310Z 
2023-01-11T22:03:10.3584509Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3584573Z Ran 1 test in 2.842s
2023-01-11T22:03:10.3584578Z 
2023-01-11T22:03:10.3584640Z OK
2023-01-11T22:03:10.3584646Z 
2023-01-11T22:03:10.3584731Z Generating XML reports...
2023-01-11T22:03:10.3585076Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215435.xml
2023-01-11T22:03:10.3585350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3585478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3585815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3586045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3586351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwnsd20ui
2023-01-11T22:03:10.3586694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwnsd20ui/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3586998Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3587005Z 
2023-01-11T22:03:10.3587079Z Running tests...
2023-01-11T22:03:10.3587276Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3587512Z   test_self_remote_rref_as_self_remote_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9774
2023-01-11T22:03:10.3587672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9775
2023-01-11T22:03:10.3587827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9776
2023-01-11T22:03:10.3587979Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9777
2023-01-11T22:03:10.3588244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3588368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3588649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3588785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3588971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wvglsaf
2023-01-11T22:03:10.3589218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wvglsaf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3589517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3589642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3589911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3590048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3590232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25dmumu7
2023-01-11T22:03:10.3590428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25dmumu7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3590702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3590828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3591112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3591251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3591439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppu2nxpgt
2023-01-11T22:03:10.3591620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppu2nxpgt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3591893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3592016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3592295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3592430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3592616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12ls2tgn
2023-01-11T22:03:10.3592812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12ls2tgn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3592977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3593141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3593292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3593455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3593522Z ok (2.741s)
2023-01-11T22:03:10.3593528Z 
2023-01-11T22:03:10.3593779Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3593900Z Ran 1 test in 2.741s
2023-01-11T22:03:10.3593908Z 
2023-01-11T22:03:10.3594012Z OK
2023-01-11T22:03:10.3594019Z 
2023-01-11T22:03:10.3594148Z Generating XML reports...
2023-01-11T22:03:10.3594493Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215440.xml
2023-01-11T22:03:10.3594756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3594881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3595162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3595300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3595523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5l14mku
2023-01-11T22:03:10.3595856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5l14mku/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3596234Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3596283Z 
2023-01-11T22:03:10.3596350Z Running tests...
2023-01-11T22:03:10.3596582Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3596817Z   test_self_remote_rref_as_self_rpc_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9997
2023-01-11T22:03:10.3596973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9998
2023-01-11T22:03:10.3597128Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9999
2023-01-11T22:03:10.3597282Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10000
2023-01-11T22:03:10.3597561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3597686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3597972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3598097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3598284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9d8_jzfs
2023-01-11T22:03:10.3598479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9d8_jzfs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3598754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3598951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3599236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3599372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3599558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgr6_sksc
2023-01-11T22:03:10.3599741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgr6_sksc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3600016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3600142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3600424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3600561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3600747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7b644c7
2023-01-11T22:03:10.3600942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7b644c7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3601217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3601343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3601611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3601747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3601935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5fg5vgp
2023-01-11T22:03:10.3602171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5fg5vgp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3602451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3602735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3603017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3603354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3603454Z ok (2.751s)
2023-01-11T22:03:10.3603478Z 
2023-01-11T22:03:10.3603887Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3604023Z Ran 1 test in 2.752s
2023-01-11T22:03:10.3604031Z 
2023-01-11T22:03:10.3604138Z OK
2023-01-11T22:03:10.3604146Z 
2023-01-11T22:03:10.3604305Z Generating XML reports...
2023-01-11T22:03:10.3604911Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215445.xml
2023-01-11T22:03:10.3605263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3605389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3605782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3605992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3606185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw34y6tp2
2023-01-11T22:03:10.3606453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw34y6tp2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3606920Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3606930Z 
2023-01-11T22:03:10.3607060Z Running tests...
2023-01-11T22:03:10.3607309Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3607529Z   test_send_to_rank (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10220
2023-01-11T22:03:10.3607687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10221
2023-01-11T22:03:10.3607843Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10222
2023-01-11T22:03:10.3607988Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10223
2023-01-11T22:03:10.3608269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3608394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3608679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3608816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3609005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2hc0ir0
2023-01-11T22:03:10.3609201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2hc0ir0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3609472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3609585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3609869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3610002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3610186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgsukh_0g
2023-01-11T22:03:10.3610378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgsukh_0g/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3610647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3610771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3611048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3611210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3611442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp633_vukj
2023-01-11T22:03:10.3611666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp633_vukj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3611942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3612067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3612345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3612480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3612667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5pgju1e
2023-01-11T22:03:10.3612862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5pgju1e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3613017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3613185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3613348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3613511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3613577Z ok (2.645s)
2023-01-11T22:03:10.3613583Z 
2023-01-11T22:03:10.3613780Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3613859Z Ran 1 test in 2.645s
2023-01-11T22:03:10.3613865Z 
2023-01-11T22:03:10.3613926Z OK
2023-01-11T22:03:10.3613931Z 
2023-01-11T22:03:10.3614013Z Generating XML reports...
2023-01-11T22:03:10.3614340Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215449.xml
2023-01-11T22:03:10.3614715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3614849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3615131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3615266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3615447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ly45zzj
2023-01-11T22:03:10.3615638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ly45zzj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3615891Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3615896Z 
2023-01-11T22:03:10.3615957Z Running tests...
2023-01-11T22:03:10.3616153Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3616392Z   test_server_process_global_profiler (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10443
2023-01-11T22:03:10.3616551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10444
2023-01-11T22:03:10.3616705Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10445
2023-01-11T22:03:10.3616857Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10446
2023-01-11T22:03:10.3617136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3617259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3617537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3617661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3617885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56ueh766
2023-01-11T22:03:10.3618108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56ueh766/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3618385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3618509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3618791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3618928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3619114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkob7ov2e
2023-01-11T22:03:10.3619324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkob7ov2e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3619793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3620005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3620325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3620459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3620642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1qwz117k
2023-01-11T22:03:10.3620838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1qwz117k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3621109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3621232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3621500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3621638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3621977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4m6k41h7
2023-01-11T22:03:10.3622176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4m6k41h7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3622342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3622504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3622667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3622830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3622884Z ok (2.755s)
2023-01-11T22:03:10.3622903Z 
2023-01-11T22:03:10.3623094Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3623171Z Ran 1 test in 2.755s
2023-01-11T22:03:10.3623177Z 
2023-01-11T22:03:10.3623240Z OK
2023-01-11T22:03:10.3623245Z 
2023-01-11T22:03:10.3623331Z Generating XML reports...
2023-01-11T22:03:10.3623675Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215454.xml
2023-01-11T22:03:10.3623951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3624079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3624435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3624645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3624957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6jwf2cl
2023-01-11T22:03:10.3625247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6jwf2cl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3625554Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3625561Z 
2023-01-11T22:03:10.3625636Z Running tests...
2023-01-11T22:03:10.3625834Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3626075Z   test_set_and_get_default_rpc_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10666
2023-01-11T22:03:10.3626248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10667
2023-01-11T22:03:10.3626522Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10668
2023-01-11T22:03:10.3626787Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10669
2023-01-11T22:03:10.3627276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3627419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3627709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3627844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3628031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpszwjk3eq
2023-01-11T22:03:10.3628307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3628431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3628700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3628837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3629021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0l8yh7r6
2023-01-11T22:03:10.3629220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpszwjk3eq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3629413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0l8yh7r6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3629684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3629808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3630088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3630284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3630459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpym15set7
2023-01-11T22:03:10.3630652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpym15set7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3630930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3631053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3631334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3631468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3631651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0myu8l2l
2023-01-11T22:03:10.3631843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0myu8l2l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3631996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3632161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3632364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3632584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3632655Z ok (2.644s)
2023-01-11T22:03:10.3632660Z 
2023-01-11T22:03:10.3632861Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3632941Z Ran 1 test in 2.645s
2023-01-11T22:03:10.3632946Z 
2023-01-11T22:03:10.3633009Z OK
2023-01-11T22:03:10.3633013Z 
2023-01-11T22:03:10.3633098Z Generating XML reports...
2023-01-11T22:03:10.3633428Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215458.xml
2023-01-11T22:03:10.3633702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3633830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3634170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3634309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3634494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrqm35h6
2023-01-11T22:03:10.3634689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrqm35h6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3634946Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3634952Z 
2023-01-11T22:03:10.3635023Z Running tests...
2023-01-11T22:03:10.3635207Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3635428Z   test_shutdown_errors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10889
2023-01-11T22:03:10.3635587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10890
2023-01-11T22:03:10.3635744Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10891
2023-01-11T22:03:10.3635898Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10892
2023-01-11T22:03:10.3636173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3636299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3636579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3636702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3636890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gavg1wu
2023-01-11T22:03:10.3637087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gavg1wu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3637364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3637486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3637768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3637903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3638089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj0guhlg1
2023-01-11T22:03:10.3638273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj0guhlg1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3638544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3638667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3639016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3639195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3639408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsm0wk9w9
2023-01-11T22:03:10.3639606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsm0wk9w9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3639962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3640170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3640645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3640875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3641200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpve4uchek
2023-01-11T22:03:10.3641542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpve4uchek/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3641827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3642118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3642397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3642684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3642987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.3643279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.3643575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.3643874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.3644411Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3644946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3645456Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3645879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3646165Z [W tensorpipe_agent.cpp:624] RPC agent for worker2 won't send response to request #4 to worker0, as the agent is shutting down
2023-01-11T22:03:10.3646448Z [W tensorpipe_agent.cpp:624] RPC agent for worker3 won't send response to request #3 to worker0, as the agent is shutting down
2023-01-11T22:03:10.3646707Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3646986Z [W tensorpipe_agent.cpp:624] RPC agent for worker1 won't send response to request #5 to worker0, as the agent is shutting down
2023-01-11T22:03:10.3647256Z [W tensorpipe_agent.cpp:939] RPC agent for worker0 encountered error when reading incoming response from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3647520Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3647783Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3648141Z [W tensorpipe_agent.cpp:939] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3648433Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3648912Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3649395Z [W tensorpipe_agent.cpp:939] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3649699Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3650243Z ERROR:torch.distributed.rpc.api:Failed to respond to 'Shutdown Proceed' in time, got error Followers ['worker3', 'worker2', 'worker1'] timed out in _all_gather after 0.00 seconds. The first exception is eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3650340Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3650460Z RuntimeError('simulation')
2023-01-11T22:03:10.3650540Z Traceback (most recent call last):
2023-01-11T22:03:10.3650806Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3650943Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3651232Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4561, in raise_error
2023-01-11T22:03:10.3651367Z     raise RuntimeError('simulation')
2023-01-11T22:03:10.3651456Z RuntimeError: simulation
2023-01-11T22:03:10.3651464Z 
2023-01-11T22:03:10.3651556Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3651673Z RuntimeError('simulation')
2023-01-11T22:03:10.3651752Z Traceback (most recent call last):
2023-01-11T22:03:10.3652014Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3652148Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3652431Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4561, in raise_error
2023-01-11T22:03:10.3652562Z     raise RuntimeError('simulation')
2023-01-11T22:03:10.3652644Z RuntimeError: simulation
2023-01-11T22:03:10.3652649Z 
2023-01-11T22:03:10.3652738Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3652841Z RuntimeError('simulation')
2023-01-11T22:03:10.3652931Z Traceback (most recent call last):
2023-01-11T22:03:10.3653188Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3653322Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3653602Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4561, in raise_error
2023-01-11T22:03:10.3653732Z     raise RuntimeError('simulation')
2023-01-11T22:03:10.3653814Z RuntimeError: simulation
2023-01-11T22:03:10.3653820Z 
2023-01-11T22:03:10.3653886Z ok (2.764s)
2023-01-11T22:03:10.3653891Z 
2023-01-11T22:03:10.3654078Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3654153Z Ran 1 test in 2.764s
2023-01-11T22:03:10.3654158Z 
2023-01-11T22:03:10.3654219Z OK
2023-01-11T22:03:10.3654224Z 
2023-01-11T22:03:10.3654307Z Generating XML reports...
2023-01-11T22:03:10.3654680Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215503.xml
2023-01-11T22:03:10.3655220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3655498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3655983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3656123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3656298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8lcbvt44
2023-01-11T22:03:10.3656499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8lcbvt44/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3656757Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3656763Z 
2023-01-11T22:03:10.3656837Z Running tests...
2023-01-11T22:03:10.3657037Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3657269Z   test_shutdown_followed_by_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11124
2023-01-11T22:03:10.3657432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11125
2023-01-11T22:03:10.3657587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11126
2023-01-11T22:03:10.3657740Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11127
2023-01-11T22:03:10.3658003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3658130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3658415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3658550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3658739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj54gcivq
2023-01-11T22:03:10.3658937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj54gcivq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3659212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3659336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3659607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3659743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3659926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kcoybk_
2023-01-11T22:03:10.3660119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kcoybk_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3660397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3660525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3660813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3660951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3661139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2c3e2ln
2023-01-11T22:03:10.3661323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2c3e2ln/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3661598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3661883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3662206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3662413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3662635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6s4rckp
2023-01-11T22:03:10.3662836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6s4rckp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3663001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3663155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3663321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3663484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3663555Z ok (2.650s)
2023-01-11T22:03:10.3663562Z 
2023-01-11T22:03:10.3663767Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3663845Z Ran 1 test in 2.650s
2023-01-11T22:03:10.3663850Z 
2023-01-11T22:03:10.3663914Z OK
2023-01-11T22:03:10.3663919Z 
2023-01-11T22:03:10.3664009Z Generating XML reports...
2023-01-11T22:03:10.3664354Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215508.xml
2023-01-11T22:03:10.3664741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3664944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3665388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3665528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3665716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhj2n6ew
2023-01-11T22:03:10.3665923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhj2n6ew/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3666187Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3666193Z 
2023-01-11T22:03:10.3666268Z Running tests...
2023-01-11T22:03:10.3666452Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3666671Z   test_stress_heavy_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11347
2023-01-11T22:03:10.3666830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11348
2023-01-11T22:03:10.3666988Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11349
2023-01-11T22:03:10.3667147Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11350
2023-01-11T22:03:10.3667425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3667553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3667840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3667977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3668155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkq8pqzr
2023-01-11T22:03:10.3668355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkq8pqzr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3668629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3668754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3669035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3669214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3669431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps50dzgw5
2023-01-11T22:03:10.3669626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps50dzgw5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3669883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3670007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3670408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3670638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3670948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_fcshoe
2023-01-11T22:03:10.3671169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_fcshoe/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3671453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3671577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3671856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3671976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3672159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbj1k54i
2023-01-11T22:03:10.3672350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbj1k54i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3672514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3672678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3672843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3673005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3673125Z Rank 0 finished testing 20 times in 0.09398698806762695 seconds.
2023-01-11T22:03:10.3673229Z Rank 1 finished testing 20 times in 0.09208393096923828 seconds.
2023-01-11T22:03:10.3673345Z Rank 2 finished testing 20 times in 0.08368444442749023 seconds.
2023-01-11T22:03:10.3673459Z Rank 3 finished testing 20 times in 0.05427694320678711 seconds.
2023-01-11T22:03:10.3673526Z ok (2.749s)
2023-01-11T22:03:10.3673531Z 
2023-01-11T22:03:10.3673729Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3673804Z Ran 1 test in 2.750s
2023-01-11T22:03:10.3673809Z 
2023-01-11T22:03:10.3673868Z OK
2023-01-11T22:03:10.3673873Z 
2023-01-11T22:03:10.3673956Z Generating XML reports...
2023-01-11T22:03:10.3674287Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215512.xml
2023-01-11T22:03:10.3674564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3674688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3674970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3675104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3675288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0p7gfsn
2023-01-11T22:03:10.3675483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0p7gfsn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3675739Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3675783Z 
2023-01-11T22:03:10.3675857Z Running tests...
2023-01-11T22:03:10.3676041Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3676301Z   test_stress_heavy_rpc_torchscript (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11570
2023-01-11T22:03:10.3676460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11571
2023-01-11T22:03:10.3676615Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11572
2023-01-11T22:03:10.3676769Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11573
2023-01-11T22:03:10.3677046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3677175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3677543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3677681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3677858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0kt88y1
2023-01-11T22:03:10.3678054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0kt88y1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3678331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3678454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3678739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3678947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3679136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8x93x2d
2023-01-11T22:03:10.3679333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8x93x2d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3679596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3679723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3680004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3680139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3680327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6llko7y
2023-01-11T22:03:10.3680522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6llko7y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3680794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3680922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3681203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3681326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3681510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_4t6vp8
2023-01-11T22:03:10.3681702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_4t6vp8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3681867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3682032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3682194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3682355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3682513Z Rank 1 finished testing 20 times in 0.1139993667602539 seconds.
2023-01-11T22:03:10.3682616Z Rank 2 finished testing 20 times in 0.12150764465332031 seconds.
2023-01-11T22:03:10.3682757Z Rank 0 finished testing 20 times in 0.10292649269104004 seconds.
2023-01-11T22:03:10.3682872Z Rank 3 finished testing 20 times in 0.11561083793640137 seconds.
2023-01-11T22:03:10.3682938Z ok (2.743s)
2023-01-11T22:03:10.3682944Z 
2023-01-11T22:03:10.3683143Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3683217Z Ran 1 test in 2.744s
2023-01-11T22:03:10.3683223Z 
2023-01-11T22:03:10.3683283Z OK
2023-01-11T22:03:10.3683288Z 
2023-01-11T22:03:10.3683371Z Generating XML reports...
2023-01-11T22:03:10.3683700Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215517.xml
2023-01-11T22:03:10.3684116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3684328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3684813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3685033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3685223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7pzqxs6
2023-01-11T22:03:10.3685423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7pzqxs6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3685681Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3685687Z 
2023-01-11T22:03:10.3685760Z Running tests...
2023-01-11T22:03:10.3685946Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3686169Z   test_stress_light_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11793
2023-01-11T22:03:10.3686327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11794
2023-01-11T22:03:10.3686486Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11795
2023-01-11T22:03:10.3686653Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11796
2023-01-11T22:03:10.3687099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3687228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3687512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3687638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3687822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8tqluxf
2023-01-11T22:03:10.3688129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8tqluxf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3688472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3688595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3688875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3689010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3689196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlbaioqg
2023-01-11T22:03:10.3689393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlbaioqg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3689672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3689937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3690482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3690728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3691011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9998ta6
2023-01-11T22:03:10.3691347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9998ta6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3691834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3692052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3692548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3692766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3693098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddgff3sp
2023-01-11T22:03:10.3693455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddgff3sp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3693676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3693842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3694003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3694163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3694278Z Rank 0 finished testing 1000 times in 0.27089834213256836 seconds.
2023-01-11T22:03:10.3694377Z Rank 1 finished testing 1000 times in 0.27671360969543457 seconds.
2023-01-11T22:03:10.3694485Z Rank 2 finished testing 1000 times in 0.24187731742858887 seconds.
2023-01-11T22:03:10.3694601Z Rank 3 finished testing 1000 times in 0.3051638603210449 seconds.
2023-01-11T22:03:10.3694668Z ok (2.948s)
2023-01-11T22:03:10.3694673Z 
2023-01-11T22:03:10.3694878Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3694953Z Ran 1 test in 2.949s
2023-01-11T22:03:10.3694958Z 
2023-01-11T22:03:10.3695020Z OK
2023-01-11T22:03:10.3695024Z 
2023-01-11T22:03:10.3695108Z Generating XML reports...
2023-01-11T22:03:10.3695435Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215522.xml
2023-01-11T22:03:10.3695710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3695834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3696203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3696437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3696759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81u2i_nd
2023-01-11T22:03:10.3696959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81u2i_nd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3697218Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3697224Z 
2023-01-11T22:03:10.3697295Z Running tests...
2023-01-11T22:03:10.3697483Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3697778Z   test_use_rpc_pickler (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12016
2023-01-11T22:03:10.3697935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12017
2023-01-11T22:03:10.3698089Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12018
2023-01-11T22:03:10.3698293Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12019
2023-01-11T22:03:10.3698609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3698735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3699021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3699145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3699333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr8vbz_h6
2023-01-11T22:03:10.3699528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr8vbz_h6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3699804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3699932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3700219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3700357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3700545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppooz43sa
2023-01-11T22:03:10.3700741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppooz43sa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3701002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3701127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3701408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3701566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3701989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_ade54l
2023-01-11T22:03:10.3702320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_ade54l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3702647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3702771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3703039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3703175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3703360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6nzi7ysl
2023-01-11T22:03:10.3703557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6nzi7ysl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3703725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3703892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3704056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3704218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3704287Z ok (2.341s)
2023-01-11T22:03:10.3704293Z 
2023-01-11T22:03:10.3704477Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3704555Z Ran 1 test in 2.342s
2023-01-11T22:03:10.3704561Z 
2023-01-11T22:03:10.3704623Z OK
2023-01-11T22:03:10.3704628Z 
2023-01-11T22:03:10.3704714Z Generating XML reports...
2023-01-11T22:03:10.3705058Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215527.xml
2023-01-11T22:03:10.3705628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3705910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3706378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3706502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3706691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85dv84is
2023-01-11T22:03:10.3706884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85dv84is/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3707142Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3707148Z 
2023-01-11T22:03:10.3707225Z Running tests...
2023-01-11T22:03:10.3707426Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3707660Z   test_use_rref_after_shutdown (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12071
2023-01-11T22:03:10.3707918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12072
2023-01-11T22:03:10.3708183Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12073
2023-01-11T22:03:10.3708420Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12074
2023-01-11T22:03:10.3708700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3708822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3709107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3709241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3709430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizs58erm
2023-01-11T22:03:10.3709629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizs58erm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3709905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3710016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3710296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3710430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3710616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdn0zlafe
2023-01-11T22:03:10.3710811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdn0zlafe/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3711088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3711212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3711495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3711633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3711807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqva7nzv6
2023-01-11T22:03:10.3712001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqva7nzv6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3712275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3712398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3712676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3712850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3713063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgx850z87
2023-01-11T22:03:10.3713256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgx850z87/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3713460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3713614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3713777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3713940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3714009Z ok (2.643s)
2023-01-11T22:03:10.3714014Z 
2023-01-11T22:03:10.3714216Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3714296Z Ran 1 test in 2.644s
2023-01-11T22:03:10.3714301Z 
2023-01-11T22:03:10.3714363Z OK
2023-01-11T22:03:10.3714368Z 
2023-01-11T22:03:10.3714456Z Generating XML reports...
2023-01-11T22:03:10.3714786Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215531.xml
2023-01-11T22:03:10.3715063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3715188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3715470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3715607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3715800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrri0at7
2023-01-11T22:03:10.3716000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrri0at7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3716259Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3716265Z 
2023-01-11T22:03:10.3716340Z Running tests...
2023-01-11T22:03:10.3716526Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3716747Z   test_user_rref_backward (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12294
2023-01-11T22:03:10.3716903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12295
2023-01-11T22:03:10.3717059Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12296
2023-01-11T22:03:10.3717215Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12297
2023-01-11T22:03:10.3717489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3717615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3717897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3718024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3718209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7zi0jrd
2023-01-11T22:03:10.3718406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7zi0jrd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3718681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3718805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3719185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3719359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3719546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8c22dl3y
2023-01-11T22:03:10.3719771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8c22dl3y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3720034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3720160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3720442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3720579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3720764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprv230q4a
2023-01-11T22:03:10.3720959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprv230q4a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3721241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3721367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3721635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3721772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3721956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiyxcjgis
2023-01-11T22:03:10.3722158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiyxcjgis/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3722324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3722488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3722655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3722820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3722890Z ok (2.745s)
2023-01-11T22:03:10.3722895Z 
2023-01-11T22:03:10.3723083Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3723182Z Ran 1 test in 2.745s
2023-01-11T22:03:10.3723189Z 
2023-01-11T22:03:10.3723285Z OK
2023-01-11T22:03:10.3723292Z 
2023-01-11T22:03:10.3723427Z Generating XML reports...
2023-01-11T22:03:10.3723817Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215535.xml
2023-01-11T22:03:10.3724093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3724218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3724503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3724629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3724818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9m8pi2p2
2023-01-11T22:03:10.3725014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9m8pi2p2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3725273Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3725279Z 
2023-01-11T22:03:10.3725353Z Running tests...
2023-01-11T22:03:10.3725555Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3725782Z   test_user_rrefs_confirmed (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12537
2023-01-11T22:03:10.3725939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12538
2023-01-11T22:03:10.3726142Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12539
2023-01-11T22:03:10.3726312Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12540
2023-01-11T22:03:10.3726684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3726893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3727391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3727607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3727803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp18qo8hpt
2023-01-11T22:03:10.3728082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3728212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3728483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3728622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3728808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvq5dlid9
2023-01-11T22:03:10.3729079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3729205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3729485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3729620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3729805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpla1c5_0a
2023-01-11T22:03:10.3730003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp18qo8hpt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3730184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvq5dlid9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3730374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpla1c5_0a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3730645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3730767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3731046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3731179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3731362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnl2izv8o
2023-01-11T22:03:10.3731557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnl2izv8o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3731720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3731871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3732033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3732194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3732261Z ok (2.754s)
2023-01-11T22:03:10.3732266Z 
2023-01-11T22:03:10.3732464Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3732540Z Ran 1 test in 2.754s
2023-01-11T22:03:10.3732545Z 
2023-01-11T22:03:10.3732605Z OK
2023-01-11T22:03:10.3732610Z 
2023-01-11T22:03:10.3732694Z Generating XML reports...
2023-01-11T22:03:10.3733061Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215540.xml
2023-01-11T22:03:10.3733365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3733491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3733772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3733908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3734091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4cqe75y
2023-01-11T22:03:10.3734284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4cqe75y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3734539Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3734546Z 
2023-01-11T22:03:10.3734619Z Running tests...
2023-01-11T22:03:10.3734804Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3735039Z   test_user_rrefs_confirmed_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12760
2023-01-11T22:03:10.3735195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12761
2023-01-11T22:03:10.3735351Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12762
2023-01-11T22:03:10.3735504Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12763
2023-01-11T22:03:10.3735776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3735899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3736182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3736308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3736495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprsu4rmmc
2023-01-11T22:03:10.3736690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprsu4rmmc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3736963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3737129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3737614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3737846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3738167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0c96og3u
2023-01-11T22:03:10.3738505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0c96og3u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3738954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3739165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3739661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3739888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3740209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7tfqgxv
2023-01-11T22:03:10.3740559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7tfqgxv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3741020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3741219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3741849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3742157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3742453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnppqin3g
2023-01-11T22:03:10.3742691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnppqin3g/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3742856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3743023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3743186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3743349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3743420Z ok (2.745s)
2023-01-11T22:03:10.3743427Z 
2023-01-11T22:03:10.3743623Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3743703Z Ran 1 test in 2.745s
2023-01-11T22:03:10.3743709Z 
2023-01-11T22:03:10.3743771Z OK
2023-01-11T22:03:10.3743776Z 
2023-01-11T22:03:10.3743862Z Generating XML reports...
2023-01-11T22:03:10.3744211Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215545.xml
2023-01-11T22:03:10.3744487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3744614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3744896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3745020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3745206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcd_xqzdg
2023-01-11T22:03:10.3745403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcd_xqzdg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3745661Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3745667Z 
2023-01-11T22:03:10.3745743Z Running tests...
2023-01-11T22:03:10.3745942Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3746153Z   test_wait_all (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12983
2023-01-11T22:03:10.3746314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12984
2023-01-11T22:03:10.3746467Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12985
2023-01-11T22:03:10.3746609Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12986
2023-01-11T22:03:10.3746886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3747012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3747295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3747430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3747614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyyckr4h7
2023-01-11T22:03:10.3747813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyyckr4h7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3748086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3748198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3748568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3748707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3748926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6h3gh_dq
2023-01-11T22:03:10.3749124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6h3gh_dq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3749397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3749521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3749802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3749939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3750112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6b6_0zs
2023-01-11T22:03:10.3750309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6b6_0zs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3750585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3750709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3750989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3751125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3751311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjam810pd
2023-01-11T22:03:10.3751505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjam810pd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3751671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3751826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3751990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3752151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3752218Z ok (2.643s)
2023-01-11T22:03:10.3752224Z 
2023-01-11T22:03:10.3752421Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3752497Z Ran 1 test in 2.644s
2023-01-11T22:03:10.3752503Z 
2023-01-11T22:03:10.3752563Z OK
2023-01-11T22:03:10.3752568Z 
2023-01-11T22:03:10.3752651Z Generating XML reports...
2023-01-11T22:03:10.3752978Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215550.xml
2023-01-11T22:03:10.3753251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3753377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3753660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3753795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3753980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpr3uz3c
2023-01-11T22:03:10.3754176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpr3uz3c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3754431Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3754435Z 
2023-01-11T22:03:10.3754508Z Running tests...
2023-01-11T22:03:10.3754693Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3754920Z   test_wait_all_exit_early_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13206
2023-01-11T22:03:10.3755106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13207
2023-01-11T22:03:10.3755302Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13208
2023-01-11T22:03:10.3755458Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13209
2023-01-11T22:03:10.3755733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3755857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3756136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3756259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3756443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_rob51p
2023-01-11T22:03:10.3756639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_rob51p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3756915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3757040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3757321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3757457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3757642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjwzx3j9
2023-01-11T22:03:10.3757836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjwzx3j9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3758094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3758221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3758505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3758644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3758901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvpmju9e
2023-01-11T22:03:10.3759101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvpmju9e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3759377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3759502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3759771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3759906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3760094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_c11rp6
2023-01-11T22:03:10.3760289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_c11rp6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3760454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3760619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3760783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3760946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3761125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.3761290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.3761496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.3761695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.3761999Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3762299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3762593Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3762885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3762952Z ok (2.747s)
2023-01-11T22:03:10.3762959Z 
2023-01-11T22:03:10.3763159Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3763224Z Ran 1 test in 2.748s
2023-01-11T22:03:10.3763230Z 
2023-01-11T22:03:10.3763290Z OK
2023-01-11T22:03:10.3763296Z 
2023-01-11T22:03:10.3763381Z Generating XML reports...
2023-01-11T22:03:10.3763723Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215554.xml
2023-01-11T22:03:10.3763999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3764123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3764405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3764542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3764715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_g20pgjy
2023-01-11T22:03:10.3764909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_g20pgjy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3765168Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3765175Z 
2023-01-11T22:03:10.3765247Z Running tests...
2023-01-11T22:03:10.3765443Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3765672Z   test_wait_all_exit_early_python (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13441
2023-01-11T22:03:10.3765829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13442
2023-01-11T22:03:10.3765983Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13443
2023-01-11T22:03:10.3766135Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13444
2023-01-11T22:03:10.3766398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3766522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3766806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3766941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3767127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpmudsqa
2023-01-11T22:03:10.3767322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpmudsqa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3767593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3767721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3768001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3768152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3768338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphm8o9wxc
2023-01-11T22:03:10.3768557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphm8o9wxc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3768829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3768952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3769232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3769367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3769550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcl0csqzo
2023-01-11T22:03:10.3769732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcl0csqzo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3770007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3770132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3770408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3770541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3770724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fxc1b49
2023-01-11T22:03:10.3770918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fxc1b49/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3771085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3771247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3771400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3771562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3771739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.3771915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.3772090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.3772263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.3772566Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3772857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3773155Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3773432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3773525Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3773647Z ValueError('Expected error')
2023-01-11T22:03:10.3773741Z Traceback (most recent call last):
2023-01-11T22:03:10.3774003Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3774139Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3774424Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3774516Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3774590Z ValueError: Expected error
2023-01-11T22:03:10.3774623Z 
2023-01-11T22:03:10.3774715Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3774836Z ValueError('Expected error')
2023-01-11T22:03:10.3774954Z Traceback (most recent call last):
2023-01-11T22:03:10.3775213Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3775346Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3775626Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3775717Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3775790Z ValueError: Expected error
2023-01-11T22:03:10.3775795Z 
2023-01-11T22:03:10.3775883Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3776002Z ValueError('Expected error')
2023-01-11T22:03:10.3776094Z Traceback (most recent call last):
2023-01-11T22:03:10.3776349Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3776486Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3776770Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3776850Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3776936Z ValueError: Expected error
2023-01-11T22:03:10.3776942Z 
2023-01-11T22:03:10.3777032Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3777152Z ValueError('Expected error')
2023-01-11T22:03:10.3777246Z Traceback (most recent call last):
2023-01-11T22:03:10.3777500Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3777636Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3777912Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3777993Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3778083Z ValueError: Expected error
2023-01-11T22:03:10.3778088Z 
2023-01-11T22:03:10.3778180Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3778304Z ValueError('Expected error')
2023-01-11T22:03:10.3778395Z Traceback (most recent call last):
2023-01-11T22:03:10.3778648Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3778781Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3779057Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3779136Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3779222Z ValueError: Expected error
2023-01-11T22:03:10.3779228Z 
2023-01-11T22:03:10.3779316Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3779436Z ValueError('Expected error')
2023-01-11T22:03:10.3779530Z Traceback (most recent call last):
2023-01-11T22:03:10.3779782Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3779917Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3780192Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3780272Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3780357Z ValueError: Expected error
2023-01-11T22:03:10.3780362Z 
2023-01-11T22:03:10.3780452Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3780571Z ValueError('Expected error')
2023-01-11T22:03:10.3780663Z Traceback (most recent call last):
2023-01-11T22:03:10.3780915Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3781078Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3781459Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3781553Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3781781Z ValueError: Expected error
2023-01-11T22:03:10.3781789Z 
2023-01-11T22:03:10.3781884Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3782007Z ValueError('Expected error')
2023-01-11T22:03:10.3782101Z Traceback (most recent call last):
2023-01-11T22:03:10.3782356Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3782487Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3782749Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3782842Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3782928Z ValueError: Expected error
2023-01-11T22:03:10.3782935Z 
2023-01-11T22:03:10.3783004Z ok (2.755s)
2023-01-11T22:03:10.3783009Z 
2023-01-11T22:03:10.3783210Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3783289Z Ran 1 test in 2.755s
2023-01-11T22:03:10.3783295Z 
2023-01-11T22:03:10.3783358Z OK
2023-01-11T22:03:10.3783363Z 
2023-01-11T22:03:10.3783449Z Generating XML reports...
2023-01-11T22:03:10.3783778Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215559.xml
2023-01-11T22:03:10.3784051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3784179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3784464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3784600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3784791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tbp_8tc
2023-01-11T22:03:10.3784989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tbp_8tc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3785247Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3785252Z 
2023-01-11T22:03:10.3785324Z Running tests...
2023-01-11T22:03:10.3785509Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3785750Z   test_wait_all_exit_early_script_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13676
2023-01-11T22:03:10.3785910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13677
2023-01-11T22:03:10.3786067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13678
2023-01-11T22:03:10.3786218Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13679
2023-01-11T22:03:10.3786494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3786619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3786903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3787039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3787211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1d7h_x_
2023-01-11T22:03:10.3787483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3787605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3787884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3788075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3788295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp433bkm0_
2023-01-11T22:03:10.3788492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1d7h_x_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3788681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp433bkm0_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3788945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3789069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3789350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3789486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3789671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphi34jr3n
2023-01-11T22:03:10.3789867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphi34jr3n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3790141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3790265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3790544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3790667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3790850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqhpyx3n
2023-01-11T22:03:10.3791042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqhpyx3n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3791205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3791373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3791537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3791706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3791966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.3792151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.3792379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.3792687Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3792858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.3793164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3793465Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3793757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3793826Z ok (2.750s)
2023-01-11T22:03:10.3793832Z 
2023-01-11T22:03:10.3794029Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3794092Z Ran 1 test in 2.750s
2023-01-11T22:03:10.3794110Z 
2023-01-11T22:03:10.3794158Z OK
2023-01-11T22:03:10.3794163Z 
2023-01-11T22:03:10.3794246Z Generating XML reports...
2023-01-11T22:03:10.3794587Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215604.xml
2023-01-11T22:03:10.3794911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3795066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3795352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3795488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3795676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptygzjgxm
2023-01-11T22:03:10.3795863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptygzjgxm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3796119Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3796125Z 
2023-01-11T22:03:10.3796197Z Running tests...
2023-01-11T22:03:10.3796397Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3796624Z   test_wait_all_multiple_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13911
2023-01-11T22:03:10.3796783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13912
2023-01-11T22:03:10.3796938Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13913
2023-01-11T22:03:10.3797093Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13914
2023-01-11T22:03:10.3797370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3797481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3797766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3797901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3798095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptiz4zyt6
2023-01-11T22:03:10.3798293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptiz4zyt6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3798567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3798690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3799078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3799202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3799387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe66fun8x
2023-01-11T22:03:10.3799581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe66fun8x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3799857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3799985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3800268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3800405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3800589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnzl8007e
2023-01-11T22:03:10.3800779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnzl8007e/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3801035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3801160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3801436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3801617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3801842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51bbqwgr
2023-01-11T22:03:10.3802037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51bbqwgr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3802202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3802366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3802515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3802675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3802743Z ok (2.745s)
2023-01-11T22:03:10.3802749Z 
2023-01-11T22:03:10.3802953Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3803032Z Ran 1 test in 2.745s
2023-01-11T22:03:10.3803038Z 
2023-01-11T22:03:10.3803101Z OK
2023-01-11T22:03:10.3803106Z 
2023-01-11T22:03:10.3803193Z Generating XML reports...
2023-01-11T22:03:10.3803537Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215608.xml
2023-01-11T22:03:10.3803812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3803924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3804207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3804343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3804530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe97pkx90
2023-01-11T22:03:10.3804729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe97pkx90/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3804986Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3804993Z 
2023-01-11T22:03:10.3805068Z Running tests...
2023-01-11T22:03:10.3805268Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3805482Z   test_wait_all_raise_in_body (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14134
2023-01-11T22:03:10.3805641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14135
2023-01-11T22:03:10.3805793Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14136
2023-01-11T22:03:10.3805945Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14137
2023-01-11T22:03:10.3806222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3806349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3813275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3813465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3813654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgkndx8l
2023-01-11T22:03:10.3813846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgkndx8l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3814142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3814269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3814558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3814779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3814968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rzuuh59
2023-01-11T22:03:10.3815199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rzuuh59/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3815484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3815611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3815897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3816023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3816209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5bmckhn
2023-01-11T22:03:10.3816406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5bmckhn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3816684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3816812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3817096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3817232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3817417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dd1wuxs
2023-01-11T22:03:10.3817599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dd1wuxs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3817768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3817934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3818100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3818266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3818335Z ok (2.753s)
2023-01-11T22:03:10.3818343Z 
2023-01-11T22:03:10.3818548Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3818627Z Ran 1 test in 2.754s
2023-01-11T22:03:10.3818633Z 
2023-01-11T22:03:10.3818695Z OK
2023-01-11T22:03:10.3818701Z 
2023-01-11T22:03:10.3818772Z Generating XML reports...
2023-01-11T22:03:10.3819117Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215613.xml
2023-01-11T22:03:10.3819395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3819520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3819805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3819943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3820130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx7jlrbtk
2023-01-11T22:03:10.3820327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx7jlrbtk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3820571Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3820590Z 
2023-01-11T22:03:10.3820650Z Running tests...
2023-01-11T22:03:10.3820850Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3821083Z   test_wait_all_raise_in_user_func (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14357
2023-01-11T22:03:10.3821247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14358
2023-01-11T22:03:10.3821442Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14359
2023-01-11T22:03:10.3821625Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14360
2023-01-11T22:03:10.3822086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3822212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3822483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3822624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3822813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjeuv19xk
2023-01-11T22:03:10.3823013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjeuv19xk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3823292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3823420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3823703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3823840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3824012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplftmgc9r
2023-01-11T22:03:10.3824211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplftmgc9r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3824487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3824611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3824898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3825041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3825227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2jhfrbx
2023-01-11T22:03:10.3825423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2jhfrbx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3825697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3825808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3826090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3826225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3826409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7xt5o3k
2023-01-11T22:03:10.3826608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7xt5o3k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3826778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3826945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3827110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3827272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3827354Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3827477Z ValueError('Expected error')
2023-01-11T22:03:10.3827572Z Traceback (most recent call last):
2023-01-11T22:03:10.3827836Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3827973Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3828325Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3828455Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3828533Z ValueError: Expected error
2023-01-11T22:03:10.3828554Z 
2023-01-11T22:03:10.3828632Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3828752Z ValueError('Expected error')
2023-01-11T22:03:10.3828846Z Traceback (most recent call last):
2023-01-11T22:03:10.3829107Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3829241Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3829522Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3829616Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3829689Z ValueError: Expected error
2023-01-11T22:03:10.3829697Z 
2023-01-11T22:03:10.3829789Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3829909Z ValueError('Expected error')
2023-01-11T22:03:10.3830004Z Traceback (most recent call last):
2023-01-11T22:03:10.3830262Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3830394Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3830673Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3830766Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3830839Z ValueError: Expected error
2023-01-11T22:03:10.3830845Z 
2023-01-11T22:03:10.3830936Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3831055Z ValueError('Expected error')
2023-01-11T22:03:10.3831148Z Traceback (most recent call last):
2023-01-11T22:03:10.3831401Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3831539Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3831825Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3831922Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3831994Z ValueError: Expected error
2023-01-11T22:03:10.3832000Z 
2023-01-11T22:03:10.3832069Z ok (2.649s)
2023-01-11T22:03:10.3832075Z 
2023-01-11T22:03:10.3832277Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3832354Z Ran 1 test in 2.649s
2023-01-11T22:03:10.3832359Z 
2023-01-11T22:03:10.3832421Z OK
2023-01-11T22:03:10.3832426Z 
2023-01-11T22:03:10.3832512Z Generating XML reports...
2023-01-11T22:03:10.3832857Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215618.xml
2023-01-11T22:03:10.3833134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3833245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3833531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3833671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3833861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_9fh7em
2023-01-11T22:03:10.3834061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_9fh7em/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3834319Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3834325Z 
2023-01-11T22:03:10.3834400Z Running tests...
2023-01-11T22:03:10.3834599Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3834857Z   test_wait_all_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14580
2023-01-11T22:03:10.3835030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14581
2023-01-11T22:03:10.3835190Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14582
2023-01-11T22:03:10.3835345Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14583
2023-01-11T22:03:10.3835623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3835749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3836034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3836173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3836360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjweoc6lx
2023-01-11T22:03:10.3836547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjweoc6lx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3836820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3836945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3837230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3837365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3837551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiv94zuj0
2023-01-11T22:03:10.3837749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiv94zuj0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3838030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3838159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3838429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3838566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3838750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplj_2y3ip
2023-01-11T22:03:10.3839017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplj_2y3ip/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3839295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3839419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3839700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3839839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3840014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7m1tgac
2023-01-11T22:03:10.3840212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7m1tgac/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3840377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3840543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3840709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3840872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3841148Z [W tensorpipe_agent.cpp:939] RPC agent for worker2 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3841476Z [W tensorpipe_agent.cpp:624] RPC agent for worker1 won't send response to request #3 to worker0, as the agent is shutting down
2023-01-11T22:03:10.3841794Z [W tensorpipe_agent.cpp:624] RPC agent for worker2 won't send response to request #1 to worker1, as the agent is shutting down
2023-01-11T22:03:10.3842079Z [W tensorpipe_agent.cpp:624] RPC agent for worker0 won't send response to request #1 to worker3, as the agent is shutting down
2023-01-11T22:03:10.3842348Z [W tensorpipe_agent.cpp:624] RPC agent for worker3 won't send response to request #1 to worker2, as the agent is shutting down
2023-01-11T22:03:10.3842421Z ok (3.645s)
2023-01-11T22:03:10.3842427Z 
2023-01-11T22:03:10.3842625Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3842706Z Ran 1 test in 3.645s
2023-01-11T22:03:10.3842712Z 
2023-01-11T22:03:10.3842776Z OK
2023-01-11T22:03:10.3842781Z 
2023-01-11T22:03:10.3842871Z Generating XML reports...
2023-01-11T22:03:10.3843218Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215622.xml
2023-01-11T22:03:10.3843499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3843611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3843892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3844029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3844215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5yibc3yx
2023-01-11T22:03:10.3844413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5yibc3yx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3844671Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3844679Z 
2023-01-11T22:03:10.3844754Z Running tests...
2023-01-11T22:03:10.3844953Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3845184Z   test_wait_all_with_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14803
2023-01-11T22:03:10.3845329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14804
2023-01-11T22:03:10.3845488Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14805
2023-01-11T22:03:10.3845642Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14806
2023-01-11T22:03:10.3845920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3846046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3846331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3846470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3846659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mvaw669
2023-01-11T22:03:10.3846845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mvaw669/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3847121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3847246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3847531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3847669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3847855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz14de485
2023-01-11T22:03:10.3848083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz14de485/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3848384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3848511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3848778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3848913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3849101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhcw12va
2023-01-11T22:03:10.3849298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhcw12va/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3849575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3849703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3849987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3850125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3850310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kkjwrkz
2023-01-11T22:03:10.3850493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kkjwrkz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3850663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3850827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3850997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3851159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3851257Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3851379Z ValueError('Expected error')
2023-01-11T22:03:10.3851477Z Traceback (most recent call last):
2023-01-11T22:03:10.3851726Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3851862Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3852146Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3852241Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3852329Z ValueError: Expected error
2023-01-11T22:03:10.3852335Z 
2023-01-11T22:03:10.3852427Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3852548Z ValueError('Expected error')
2023-01-11T22:03:10.3852626Z Traceback (most recent call last):
2023-01-11T22:03:10.3852887Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3853025Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3853309Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3853403Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3853490Z ValueError: Expected error
2023-01-11T22:03:10.3853495Z 
2023-01-11T22:03:10.3853586Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3853708Z ValueError('Expected error')
2023-01-11T22:03:10.3853786Z Traceback (most recent call last):
2023-01-11T22:03:10.3854044Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3854178Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3854463Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3854589Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3854676Z ValueError: Expected error
2023-01-11T22:03:10.3854681Z 
2023-01-11T22:03:10.3854845Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3854968Z ValueError('Expected error')
2023-01-11T22:03:10.3855046Z Traceback (most recent call last):
2023-01-11T22:03:10.3855307Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3855438Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3855721Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3855814Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3855901Z ValueError: Expected error
2023-01-11T22:03:10.3855908Z 
2023-01-11T22:03:10.3855999Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3856119Z ValueError('Expected error')
2023-01-11T22:03:10.3856200Z Traceback (most recent call last):
2023-01-11T22:03:10.3856456Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3856592Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3856871Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3856965Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3857052Z ValueError: Expected error
2023-01-11T22:03:10.3857057Z 
2023-01-11T22:03:10.3857148Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3857269Z ValueError('Expected error')
2023-01-11T22:03:10.3857348Z Traceback (most recent call last):
2023-01-11T22:03:10.3857602Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3857735Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3858014Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3858108Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3858196Z ValueError: Expected error
2023-01-11T22:03:10.3858201Z 
2023-01-11T22:03:10.3858292Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3858398Z ValueError('Expected error')
2023-01-11T22:03:10.3858492Z Traceback (most recent call last):
2023-01-11T22:03:10.3858745Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3858876Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3859153Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3859246Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3859331Z ValueError: Expected error
2023-01-11T22:03:10.3859339Z 
2023-01-11T22:03:10.3859430Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3859535Z ValueError('Expected error')
2023-01-11T22:03:10.3859628Z Traceback (most recent call last):
2023-01-11T22:03:10.3859884Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3860014Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3860290Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3860382Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3860470Z ValueError: Expected error
2023-01-11T22:03:10.3860475Z 
2023-01-11T22:03:10.3860565Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3860671Z ValueError('Expected error')
2023-01-11T22:03:10.3860763Z Traceback (most recent call last):
2023-01-11T22:03:10.3861015Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3861178Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3861486Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3861579Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3861783Z ValueError: Expected error
2023-01-11T22:03:10.3861790Z 
2023-01-11T22:03:10.3861892Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3862003Z ValueError('Expected error')
2023-01-11T22:03:10.3862095Z Traceback (most recent call last):
2023-01-11T22:03:10.3862348Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3862479Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3862754Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3862851Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3862936Z ValueError: Expected error
2023-01-11T22:03:10.3862941Z 
2023-01-11T22:03:10.3863032Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3863139Z ValueError('Expected error')
2023-01-11T22:03:10.3863230Z Traceback (most recent call last):
2023-01-11T22:03:10.3863482Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3863612Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3863888Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3863981Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3864067Z ValueError: Expected error
2023-01-11T22:03:10.3864072Z 
2023-01-11T22:03:10.3864149Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3864269Z ValueError('Expected error')
2023-01-11T22:03:10.3864360Z Traceback (most recent call last):
2023-01-11T22:03:10.3864613Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3864747Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3865030Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3865123Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3865209Z ValueError: Expected error
2023-01-11T22:03:10.3865216Z 
2023-01-11T22:03:10.3865292Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3865414Z ValueError('Expected error')
2023-01-11T22:03:10.3865507Z Traceback (most recent call last):
2023-01-11T22:03:10.3865763Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3865895Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3866174Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3866269Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3866356Z ValueError: Expected error
2023-01-11T22:03:10.3866362Z 
2023-01-11T22:03:10.3866439Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3866558Z ValueError('Expected error')
2023-01-11T22:03:10.3866649Z Traceback (most recent call last):
2023-01-11T22:03:10.3866907Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3867038Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3867317Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3867411Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3867482Z ValueError: Expected error
2023-01-11T22:03:10.3867502Z 
2023-01-11T22:03:10.3867579Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3867762Z ValueError('Expected error')
2023-01-11T22:03:10.3867856Z Traceback (most recent call last):
2023-01-11T22:03:10.3868147Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3868280Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3868558Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3868649Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3868720Z ValueError: Expected error
2023-01-11T22:03:10.3868738Z 
2023-01-11T22:03:10.3868813Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3868932Z ValueError('Expected error')
2023-01-11T22:03:10.3869025Z Traceback (most recent call last):
2023-01-11T22:03:10.3869279Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3869412Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3869690Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3869783Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3869856Z ValueError: Expected error
2023-01-11T22:03:10.3869862Z 
2023-01-11T22:03:10.3869951Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3870071Z ValueError('Expected error')
2023-01-11T22:03:10.3870164Z Traceback (most recent call last):
2023-01-11T22:03:10.3870419Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3870550Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3870825Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3870920Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3870995Z ValueError: Expected error
2023-01-11T22:03:10.3871001Z 
2023-01-11T22:03:10.3871091Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3871212Z ValueError('Expected error')
2023-01-11T22:03:10.3871306Z Traceback (most recent call last):
2023-01-11T22:03:10.3871559Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3871689Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3871966Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3872060Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3872131Z ValueError: Expected error
2023-01-11T22:03:10.3872136Z 
2023-01-11T22:03:10.3872228Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3872348Z ValueError('Expected error')
2023-01-11T22:03:10.3872439Z Traceback (most recent call last):
2023-01-11T22:03:10.3872692Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3872823Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3873101Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3873180Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3873268Z ValueError: Expected error
2023-01-11T22:03:10.3873274Z 
2023-01-11T22:03:10.3873362Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3873481Z ValueError('Expected error')
2023-01-11T22:03:10.3873573Z Traceback (most recent call last):
2023-01-11T22:03:10.3873827Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3873957Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3874234Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3874343Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3874429Z ValueError: Expected error
2023-01-11T22:03:10.3874434Z 
2023-01-11T22:03:10.3874552Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3874675Z ValueError('Expected error')
2023-01-11T22:03:10.3874766Z Traceback (most recent call last):
2023-01-11T22:03:10.3875015Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3875145Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3875420Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3875499Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3875583Z ValueError: Expected error
2023-01-11T22:03:10.3875589Z 
2023-01-11T22:03:10.3875678Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3875802Z ValueError('Expected error')
2023-01-11T22:03:10.3875894Z Traceback (most recent call last):
2023-01-11T22:03:10.3876145Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3876275Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3876552Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3876631Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3876716Z ValueError: Expected error
2023-01-11T22:03:10.3876721Z 
2023-01-11T22:03:10.3876812Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3876933Z ValueError('Expected error')
2023-01-11T22:03:10.3877024Z Traceback (most recent call last):
2023-01-11T22:03:10.3877273Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3877403Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3877666Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3877759Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3877845Z ValueError: Expected error
2023-01-11T22:03:10.3877849Z 
2023-01-11T22:03:10.3877940Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3878059Z ValueError('Expected error')
2023-01-11T22:03:10.3878151Z Traceback (most recent call last):
2023-01-11T22:03:10.3878404Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3878535Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3878794Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3878959Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3879048Z ValueError: Expected error
2023-01-11T22:03:10.3879053Z 
2023-01-11T22:03:10.3879146Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3879267Z ValueError('Expected error')
2023-01-11T22:03:10.3879362Z Traceback (most recent call last):
2023-01-11T22:03:10.3879615Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3879746Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3880007Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3880102Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3880187Z ValueError: Expected error
2023-01-11T22:03:10.3880192Z 
2023-01-11T22:03:10.3880281Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3880403Z ValueError('Expected error')
2023-01-11T22:03:10.3880497Z Traceback (most recent call last):
2023-01-11T22:03:10.3880757Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3880927Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3881228Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3881324Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3881409Z ValueError: Expected error
2023-01-11T22:03:10.3881415Z 
2023-01-11T22:03:10.3881506Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3881627Z ValueError('Expected error')
2023-01-11T22:03:10.3881722Z Traceback (most recent call last):
2023-01-11T22:03:10.3881997Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3882119Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3882400Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3882496Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3882582Z ValueError: Expected error
2023-01-11T22:03:10.3882588Z 
2023-01-11T22:03:10.3882679Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3882799Z ValueError('Expected error')
2023-01-11T22:03:10.3882892Z Traceback (most recent call last):
2023-01-11T22:03:10.3883148Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3883268Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3883549Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3883642Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3883729Z ValueError: Expected error
2023-01-11T22:03:10.3883734Z 
2023-01-11T22:03:10.3883828Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3883946Z ValueError('Expected error')
2023-01-11T22:03:10.3884041Z Traceback (most recent call last):
2023-01-11T22:03:10.3884296Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3884416Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3884693Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3884785Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3884873Z ValueError: Expected error
2023-01-11T22:03:10.3884878Z 
2023-01-11T22:03:10.3884967Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3885086Z ValueError('Expected error')
2023-01-11T22:03:10.3885178Z Traceback (most recent call last):
2023-01-11T22:03:10.3885429Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3885547Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3885823Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3885922Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3886008Z ValueError: Expected error
2023-01-11T22:03:10.3886013Z 
2023-01-11T22:03:10.3886101Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3886221Z ValueError('Expected error')
2023-01-11T22:03:10.3886313Z Traceback (most recent call last):
2023-01-11T22:03:10.3886551Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3886681Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3886959Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3887053Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3887138Z ValueError: Expected error
2023-01-11T22:03:10.3887176Z 
2023-01-11T22:03:10.3887267Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3887389Z ValueError('Expected error')
2023-01-11T22:03:10.3887483Z Traceback (most recent call last):
2023-01-11T22:03:10.3887750Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3887884Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3888159Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3888253Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3888338Z ValueError: Expected error
2023-01-11T22:03:10.3888343Z 
2023-01-11T22:03:10.3888433Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3888553Z ValueError('Expected error')
2023-01-11T22:03:10.3888644Z Traceback (most recent call last):
2023-01-11T22:03:10.3888881Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3889014Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3889291Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3889384Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3889470Z ValueError: Expected error
2023-01-11T22:03:10.3889475Z 
2023-01-11T22:03:10.3889564Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3889684Z ValueError('Expected error')
2023-01-11T22:03:10.3889776Z Traceback (most recent call last):
2023-01-11T22:03:10.3890012Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3890145Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3890417Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3890513Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3890598Z ValueError: Expected error
2023-01-11T22:03:10.3890603Z 
2023-01-11T22:03:10.3890694Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3890816Z ValueError('Expected error')
2023-01-11T22:03:10.3890910Z Traceback (most recent call last):
2023-01-11T22:03:10.3891155Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3891290Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3891574Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3891668Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3891753Z ValueError: Expected error
2023-01-11T22:03:10.3891758Z 
2023-01-11T22:03:10.3891851Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3891970Z ValueError('Expected error')
2023-01-11T22:03:10.3892052Z Traceback (most recent call last):
2023-01-11T22:03:10.3892306Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3892442Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3892723Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3892818Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3892905Z ValueError: Expected error
2023-01-11T22:03:10.3892910Z 
2023-01-11T22:03:10.3893001Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3893120Z ValueError('Expected error')
2023-01-11T22:03:10.3893196Z Traceback (most recent call last):
2023-01-11T22:03:10.3893450Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3893583Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3893891Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3893984Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3894099Z ValueError: Expected error
2023-01-11T22:03:10.3894105Z 
2023-01-11T22:03:10.3894197Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3894317Z ValueError('Expected error')
2023-01-11T22:03:10.3894397Z Traceback (most recent call last):
2023-01-11T22:03:10.3894648Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3894782Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3895059Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3895150Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3895235Z ValueError: Expected error
2023-01-11T22:03:10.3895240Z 
2023-01-11T22:03:10.3895334Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3895453Z ValueError('Expected error')
2023-01-11T22:03:10.3895532Z Traceback (most recent call last):
2023-01-11T22:03:10.3895785Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3895916Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3896194Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3896286Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3896375Z ValueError: Expected error
2023-01-11T22:03:10.3896381Z 
2023-01-11T22:03:10.3896471Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3896590Z ValueError('Expected error')
2023-01-11T22:03:10.3896669Z Traceback (most recent call last):
2023-01-11T22:03:10.3896918Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3897051Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3897328Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3897420Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3897505Z ValueError: Expected error
2023-01-11T22:03:10.3897511Z 
2023-01-11T22:03:10.3897580Z ok (2.745s)
2023-01-11T22:03:10.3897586Z 
2023-01-11T22:03:10.3897786Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3897849Z Ran 1 test in 2.745s
2023-01-11T22:03:10.3897854Z 
2023-01-11T22:03:10.3897917Z OK
2023-01-11T22:03:10.3897922Z 
2023-01-11T22:03:10.3898013Z Generating XML reports...
2023-01-11T22:03:10.3898358Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215628.xml
2023-01-11T22:03:10.3898635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3898764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3899050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3899188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3899362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptv3dxq1h
2023-01-11T22:03:10.3899564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptv3dxq1h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3899819Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3899824Z 
2023-01-11T22:03:10.3899899Z Running tests...
2023-01-11T22:03:10.3900099Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3900336Z   test_wait_all_with_partial_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15026
2023-01-11T22:03:10.3900531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15027
2023-01-11T22:03:10.3900727Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15028
2023-01-11T22:03:10.3900884Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15029
2023-01-11T22:03:10.3901152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3901278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3901569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3901836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3902026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1_t038v
2023-01-11T22:03:10.3902224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1_t038v/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3902502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3902628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3902898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3903039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3903225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqln3x4s
2023-01-11T22:03:10.3903422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqln3x4s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3903694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3903822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3904107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3904247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3904433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0kxp7ul
2023-01-11T22:03:10.3904692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3904819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3905098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3905234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3905421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqy84d4h2
2023-01-11T22:03:10.3905618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0kxp7ul/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3905812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqy84d4h2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3905979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3906131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3906299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3906461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3906556Z On WorkerInfo(id=0, name=worker0):
2023-01-11T22:03:10.3906677Z ValueError('Expected error')
2023-01-11T22:03:10.3906771Z Traceback (most recent call last):
2023-01-11T22:03:10.3907030Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3907233Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3907542Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3907643Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3907729Z ValueError: Expected error
2023-01-11T22:03:10.3907738Z 
2023-01-11T22:03:10.3907830Z On WorkerInfo(id=1, name=worker1):
2023-01-11T22:03:10.3907951Z ValueError('Expected error')
2023-01-11T22:03:10.3908043Z Traceback (most recent call last):
2023-01-11T22:03:10.3908301Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3908436Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3908700Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3908796Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3908882Z ValueError: Expected error
2023-01-11T22:03:10.3908887Z 
2023-01-11T22:03:10.3908978Z On WorkerInfo(id=2, name=worker2):
2023-01-11T22:03:10.3909099Z ValueError('Expected error')
2023-01-11T22:03:10.3909190Z Traceback (most recent call last):
2023-01-11T22:03:10.3909446Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3909578Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3909841Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3909933Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3910019Z ValueError: Expected error
2023-01-11T22:03:10.3910024Z 
2023-01-11T22:03:10.3910116Z On WorkerInfo(id=3, name=worker3):
2023-01-11T22:03:10.3910238Z ValueError('Expected error')
2023-01-11T22:03:10.3910329Z Traceback (most recent call last):
2023-01-11T22:03:10.3910584Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T22:03:10.3910702Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T22:03:10.3910977Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2023-01-11T22:03:10.3911069Z     raise ValueError(expected_err)
2023-01-11T22:03:10.3911153Z ValueError: Expected error
2023-01-11T22:03:10.3911158Z 
2023-01-11T22:03:10.3911226Z ok (2.739s)
2023-01-11T22:03:10.3911231Z 
2023-01-11T22:03:10.3911432Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3911508Z Ran 1 test in 2.740s
2023-01-11T22:03:10.3911513Z 
2023-01-11T22:03:10.3911574Z OK
2023-01-11T22:03:10.3911579Z 
2023-01-11T22:03:10.3911664Z Generating XML reports...
2023-01-11T22:03:10.3911995Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215632.xml
2023-01-11T22:03:10.3912276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3912403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3912689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3912827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3913014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpes9udaby
2023-01-11T22:03:10.3913213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpes9udaby/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3913471Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3913513Z 
2023-01-11T22:03:10.3913590Z Running tests...
2023-01-11T22:03:10.3913779Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3914034Z   test_wait_all_workers_dense (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15249
2023-01-11T22:03:10.3914193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15250
2023-01-11T22:03:10.3914351Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15251
2023-01-11T22:03:10.3914506Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15252
2023-01-11T22:03:10.3914785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3914951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3915233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3915359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3915548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpik3d0mb6
2023-01-11T22:03:10.3915744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpik3d0mb6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3916019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3916145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3916425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3916563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3916749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmitrqwy5
2023-01-11T22:03:10.3916942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmitrqwy5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3917204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3917328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3917609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3917744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3917927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptybhky0m
2023-01-11T22:03:10.3918124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptybhky0m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3918395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3918522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3918788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3919006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3919194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa34l7evq
2023-01-11T22:03:10.3919390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa34l7evq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3919556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3919721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3919885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3920047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3920265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.3920469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.3920776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3920951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.3921123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.3921420Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3921718Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3922009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3922285Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3922553Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3922811Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3923056Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3923317Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3923583Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3923653Z ok (2.951s)
2023-01-11T22:03:10.3923660Z 
2023-01-11T22:03:10.3923865Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3923943Z Ran 1 test in 2.952s
2023-01-11T22:03:10.3923949Z 
2023-01-11T22:03:10.3924012Z OK
2023-01-11T22:03:10.3924016Z 
2023-01-11T22:03:10.3924102Z Generating XML reports...
2023-01-11T22:03:10.3924446Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215637.xml
2023-01-11T22:03:10.3924722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3924837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3925121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3925259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3925447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi4s_7azw
2023-01-11T22:03:10.3925647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi4s_7azw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3925902Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3925907Z 
2023-01-11T22:03:10.3925982Z Running tests...
2023-01-11T22:03:10.3926181Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3926395Z   test_wait_all_workers_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15484
2023-01-11T22:03:10.3926594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15485
2023-01-11T22:03:10.3926779Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15486
2023-01-11T22:03:10.3926934Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15487
2023-01-11T22:03:10.3927214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3927340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3927625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3927764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3927955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0o0czxa
2023-01-11T22:03:10.3928142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0o0czxa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3928418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3928542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3928824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3928962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3929148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptm7h41sk
2023-01-11T22:03:10.3929344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptm7h41sk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3929618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3929732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3930015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3930154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3930341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcodbywo8
2023-01-11T22:03:10.3930536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcodbywo8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3930810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3930936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3931215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3931350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3931524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprgv4d3ej
2023-01-11T22:03:10.3931721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprgv4d3ej/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3931890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3932055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3932220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3932384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3932562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.3932739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.3932948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.3933136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.3933443Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3933741Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3934031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3934326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3934595Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3934871Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3935129Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3935391Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3935641Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3935905Z [W tensorpipe_agent.cpp:939] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3936158Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3936424Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #4 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3936711Z [W tensorpipe_agent.cpp:624] RPC agent for worker3 won't send response to request #3 to worker0, as the agent is shutting down
2023-01-11T22:03:10.3936977Z [W tensorpipe_agent.cpp:678] RPC agent for worker0 encountered error when sending response to request #1 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3937237Z [W tensorpipe_agent.cpp:678] RPC agent for worker0 encountered error when sending response to request #1 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3937497Z [W tensorpipe_agent.cpp:678] RPC agent for worker0 encountered error when sending response to request #1 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3937568Z ok (5.845s)
2023-01-11T22:03:10.3937574Z 
2023-01-11T22:03:10.3937772Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3937850Z Ran 1 test in 5.845s
2023-01-11T22:03:10.3937856Z 
2023-01-11T22:03:10.3937917Z OK
2023-01-11T22:03:10.3937922Z 
2023-01-11T22:03:10.3938010Z Generating XML reports...
2023-01-11T22:03:10.3938336Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215642.xml
2023-01-11T22:03:10.3938614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3938774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3939088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3939227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3939417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskxvw7w0
2023-01-11T22:03:10.3939617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskxvw7w0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3939873Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3939878Z 
2023-01-11T22:03:10.3939956Z Running tests...
2023-01-11T22:03:10.3940143Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3940375Z   test_wait_all_workers_twice_dense (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15719
2023-01-11T22:03:10.3940536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15720
2023-01-11T22:03:10.3940696Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15721
2023-01-11T22:03:10.3940851Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15722
2023-01-11T22:03:10.3941128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3941256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3941539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3941847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3942040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpugy5z5vd
2023-01-11T22:03:10.3942243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpugy5z5vd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3942523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3942649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3942931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3943065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3943250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_m6sfny1
2023-01-11T22:03:10.3943443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_m6sfny1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3943702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3943831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3944114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3944253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3944437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpweahu5vk
2023-01-11T22:03:10.3944633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpweahu5vk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3944906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3945031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3945296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3945431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3945679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pewpy3q
2023-01-11T22:03:10.3945918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pewpy3q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3946086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3946254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3946417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3946584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3946763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.3946927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.3947106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.3947284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.3947590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3947890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3948184Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3948476Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.3948743Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3949012Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3949271Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.3949524Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3949788Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3950054Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.3950126Z ok (2.954s)
2023-01-11T22:03:10.3950134Z 
2023-01-11T22:03:10.3950338Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3950416Z Ran 1 test in 2.955s
2023-01-11T22:03:10.3950422Z 
2023-01-11T22:03:10.3950485Z OK
2023-01-11T22:03:10.3950490Z 
2023-01-11T22:03:10.3950575Z Generating XML reports...
2023-01-11T22:03:10.3950917Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215650.xml
2023-01-11T22:03:10.3951195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3951307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3951589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3951761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3951984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqjilngl
2023-01-11T22:03:10.3952185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqjilngl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3952444Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3952450Z 
2023-01-11T22:03:10.3952526Z Running tests...
2023-01-11T22:03:10.3952723Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3952922Z   test_worker_id (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15954
2023-01-11T22:03:10.3953084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15955
2023-01-11T22:03:10.3953239Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15956
2023-01-11T22:03:10.3953399Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15957
2023-01-11T22:03:10.3953676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3953801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3954085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3954222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3954408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdnbsjnue
2023-01-11T22:03:10.3954593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdnbsjnue/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3954873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3955001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3955285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3955424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3955610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0f9wk28
2023-01-11T22:03:10.3955804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0f9wk28/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3956079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3956190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3956474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3956613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3956803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf61d2hch
2023-01-11T22:03:10.3957003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf61d2hch/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3957276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3957402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3957686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3957822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3957995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqzquaa5
2023-01-11T22:03:10.3958192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqzquaa5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3958428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3958620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3958785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3959024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3959097Z ok (2.737s)
2023-01-11T22:03:10.3959103Z 
2023-01-11T22:03:10.3959309Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3959372Z Ran 1 test in 2.737s
2023-01-11T22:03:10.3959393Z 
2023-01-11T22:03:10.3959442Z OK
2023-01-11T22:03:10.3959447Z 
2023-01-11T22:03:10.3959533Z Generating XML reports...
2023-01-11T22:03:10.3959886Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215655.xml
2023-01-11T22:03:10.3960164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3960292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3960577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3960713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3960899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpri8s14ft
2023-01-11T22:03:10.3961083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpri8s14ft/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3961342Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3961348Z 
2023-01-11T22:03:10.3961421Z Running tests...
2023-01-11T22:03:10.3961622Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3961851Z   test_worker_info_pickle (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16177
2023-01-11T22:03:10.3962011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16178
2023-01-11T22:03:10.3962167Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16179
2023-01-11T22:03:10.3962321Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16180
2023-01-11T22:03:10.3962596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3962709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3962993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3963130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3963318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_1uvnce
2023-01-11T22:03:10.3963513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_1uvnce/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3963790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3963914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3964193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3964315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3964499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0a7pc5we
2023-01-11T22:03:10.3964770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3964896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3965212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3965373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3965559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjqbkhjc
2023-01-11T22:03:10.3965755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0a7pc5we/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3965948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjqbkhjc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3966207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3966332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3966615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3966752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3966941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpks2i2mev
2023-01-11T22:03:10.3967134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpks2i2mev/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3967300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3967467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3967616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3967780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3967850Z ok (2.740s)
2023-01-11T22:03:10.3967856Z 
2023-01-11T22:03:10.3968055Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3968136Z Ran 1 test in 2.740s
2023-01-11T22:03:10.3968141Z 
2023-01-11T22:03:10.3968203Z OK
2023-01-11T22:03:10.3968208Z 
2023-01-11T22:03:10.3968328Z Generating XML reports...
2023-01-11T22:03:10.3968672Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215659.xml
2023-01-11T22:03:10.3968946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3969058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3969342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3969478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3969663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpm3v2ry
2023-01-11T22:03:10.3969858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpm3v2ry/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3970116Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3970123Z 
2023-01-11T22:03:10.3970198Z Running tests...
2023-01-11T22:03:10.3970397Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3970602Z   test_world_size_one (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16400
2023-01-11T22:03:10.3970762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16401
2023-01-11T22:03:10.3970917Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16402
2023-01-11T22:03:10.3971072Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16403
2023-01-11T22:03:10.3971349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3971508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3971818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3971957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3972142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbk_giv0c
2023-01-11T22:03:10.3972324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbk_giv0c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3972597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3972721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3973003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3973137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3973325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg072wp6c
2023-01-11T22:03:10.3973522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg072wp6c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3973795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3973906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3974187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3974323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3974507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrxrelek
2023-01-11T22:03:10.3974703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrxrelek/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3974975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3975101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3975384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3975519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3975692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmp88jib2
2023-01-11T22:03:10.3975886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmp88jib2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3976051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3976218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3976383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3976545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3976616Z ok (2.342s)
2023-01-11T22:03:10.3976623Z 
2023-01-11T22:03:10.3976821Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3976885Z Ran 1 test in 2.342s
2023-01-11T22:03:10.3976905Z 
2023-01-11T22:03:10.3976953Z OK
2023-01-11T22:03:10.3976958Z 
2023-01-11T22:03:10.3977045Z Generating XML reports...
2023-01-11T22:03:10.3977388Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215704.xml
2023-01-11T22:03:10.3977662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3977787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3978067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3978242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3978456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvykn0hx0
2023-01-11T22:03:10.3978640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvykn0hx0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3978896Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3978902Z 
2023-01-11T22:03:10.3978976Z Running tests...
2023-01-11T22:03:10.3979174Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3979392Z   test_wrong_types (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16497
2023-01-11T22:03:10.3979549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16498
2023-01-11T22:03:10.3979707Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16499
2023-01-11T22:03:10.3979864Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16500
2023-01-11T22:03:10.3980139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3980249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3980531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3980666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3980851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8eyzmk1q
2023-01-11T22:03:10.3981123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3981250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3981532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3981786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3981974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0v0jcb5c
2023-01-11T22:03:10.3982171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8eyzmk1q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3982365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0v0jcb5c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3982643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3982768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3983146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3983288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3983477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpviliffa3
2023-01-11T22:03:10.3983673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpviliffa3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3983936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3984064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3984346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3984482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3984668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgglnv8x
2023-01-11T22:03:10.3984924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgglnv8x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3985131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3985300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3985450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3985618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3985690Z ok (2.353s)
2023-01-11T22:03:10.3985696Z 
2023-01-11T22:03:10.3985900Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3985978Z Ran 1 test in 2.354s
2023-01-11T22:03:10.3985983Z 
2023-01-11T22:03:10.3986045Z OK
2023-01-11T22:03:10.3986050Z 
2023-01-11T22:03:10.3986135Z Generating XML reports...
2023-01-11T22:03:10.3986476Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215708.xml
2023-01-11T22:03:10.3986759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3986873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3987156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3987294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3987480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9q8g3uq
2023-01-11T22:03:10.3987675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9q8g3uq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3987934Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3987939Z 
2023-01-11T22:03:10.3988016Z Running tests...
2023-01-11T22:03:10.3988215Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3988499Z   test_backward_different_dtypes_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16552
2023-01-11T22:03:10.3988658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16553
2023-01-11T22:03:10.3988815Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16554
2023-01-11T22:03:10.3988971Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16555
2023-01-11T22:03:10.3989247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3989371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3989655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3989794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3989984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprihlcpgo
2023-01-11T22:03:10.3990170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprihlcpgo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3990444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3990569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3990853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3990990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3991173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72id8u8y
2023-01-11T22:03:10.3991367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72id8u8y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3991712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3991840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3992108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3992242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3992426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplt1_9c4m
2023-01-11T22:03:10.3992618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplt1_9c4m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3992888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3993014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3993301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3993438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3993679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3sqt0toc
2023-01-11T22:03:10.3993948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3sqt0toc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3994124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.3994289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.3994450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.3994614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.3994687Z ok (2.739s)
2023-01-11T22:03:10.3994693Z 
2023-01-11T22:03:10.3994898Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3994960Z Ran 1 test in 2.740s
2023-01-11T22:03:10.3994982Z 
2023-01-11T22:03:10.3995032Z OK
2023-01-11T22:03:10.3995037Z 
2023-01-11T22:03:10.3995123Z Generating XML reports...
2023-01-11T22:03:10.3995557Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215713.xml
2023-01-11T22:03:10.3995837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3995967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3996247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3996384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3996575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkthfc62m
2023-01-11T22:03:10.3996759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkthfc62m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3997017Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.3997022Z 
2023-01-11T22:03:10.3997096Z Running tests...
2023-01-11T22:03:10.3997295Z ----------------------------------------------------------------------
2023-01-11T22:03:10.3997592Z   test_backward_multiple_round_trips_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16807
2023-01-11T22:03:10.3997754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16808
2023-01-11T22:03:10.3997910Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16809
2023-01-11T22:03:10.3998108Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16810
2023-01-11T22:03:10.3998414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3998527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.3998812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.3999026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.3999216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3uyswr53
2023-01-11T22:03:10.3999415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3uyswr53/_remote_module_non_scriptable.py
2023-01-11T22:03:10.3999694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.3999819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4000105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4000229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4000417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqwl7iuu
2023-01-11T22:03:10.4000618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqwl7iuu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4000894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4001019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4001302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4001442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4001632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeahcu5ct
2023-01-11T22:03:10.4001831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeahcu5ct/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4002094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4002219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4002499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4002634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4002819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpza72lzu4
2023-01-11T22:03:10.4003014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpza72lzu4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4003181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4003349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4003515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4003665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4003735Z ok (2.848s)
2023-01-11T22:03:10.4003740Z 
2023-01-11T22:03:10.4003939Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4004016Z Ran 1 test in 2.848s
2023-01-11T22:03:10.4004021Z 
2023-01-11T22:03:10.4004083Z OK
2023-01-11T22:03:10.4004088Z 
2023-01-11T22:03:10.4004174Z Generating XML reports...
2023-01-11T22:03:10.4004601Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215717.xml
2023-01-11T22:03:10.4004877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4005021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4005333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4005472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4005661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2cpjjg9
2023-01-11T22:03:10.4005857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2cpjjg9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4006115Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4006121Z 
2023-01-11T22:03:10.4006196Z Running tests...
2023-01-11T22:03:10.4006395Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4006692Z   test_backward_no_grad_on_tensor_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17107
2023-01-11T22:03:10.4006840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17108
2023-01-11T22:03:10.4006998Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17109
2023-01-11T22:03:10.4007153Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17110
2023-01-11T22:03:10.4007431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4007556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4007840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4007978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4008169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwikmpx90
2023-01-11T22:03:10.4008432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4008558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4008840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4008977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4009164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoccplzy6
2023-01-11T22:03:10.4009361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwikmpx90/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4009556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoccplzy6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4009830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4009959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4010229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4010366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4010550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmm3htd11
2023-01-11T22:03:10.4010745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmm3htd11/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4011021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4011146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4011428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4011594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4011808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjx0jyo_c
2023-01-11T22:03:10.4011991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjx0jyo_c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4012156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4012321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4012483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4012647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4012718Z ok (2.747s)
2023-01-11T22:03:10.4012723Z 
2023-01-11T22:03:10.4012923Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4013003Z Ran 1 test in 2.747s
2023-01-11T22:03:10.4013009Z 
2023-01-11T22:03:10.4013057Z OK
2023-01-11T22:03:10.4013063Z 
2023-01-11T22:03:10.4013151Z Generating XML reports...
2023-01-11T22:03:10.4013577Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215722.xml
2023-01-11T22:03:10.4013854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4013980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4014266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4014403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4014589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjo1rykw
2023-01-11T22:03:10.4014786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjo1rykw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4015031Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4015037Z 
2023-01-11T22:03:10.4015112Z Running tests...
2023-01-11T22:03:10.4015315Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4015603Z   test_backward_rref_multi_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17362
2023-01-11T22:03:10.4015764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17363
2023-01-11T22:03:10.4015921Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17364
2023-01-11T22:03:10.4016076Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17365
2023-01-11T22:03:10.4016354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4016467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4016751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4016890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4017075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvppf1hzc
2023-01-11T22:03:10.4017271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvppf1hzc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4017546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4017670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4017955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4018123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4018322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpze4enm4m
2023-01-11T22:03:10.4018523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpze4enm4m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4018794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4018918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4019200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4019335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4019520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57pczrsf
2023-01-11T22:03:10.4019717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57pczrsf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4019995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4020107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4020392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4020528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4020713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp888ddxzc
2023-01-11T22:03:10.4020907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp888ddxzc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4021073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4021237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4021402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4021553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4021623Z ok (2.748s)
2023-01-11T22:03:10.4021628Z 
2023-01-11T22:03:10.4021960Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4022042Z Ran 1 test in 2.748s
2023-01-11T22:03:10.4022048Z 
2023-01-11T22:03:10.4022110Z OK
2023-01-11T22:03:10.4022115Z 
2023-01-11T22:03:10.4022199Z Generating XML reports...
2023-01-11T22:03:10.4022629Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215727.xml
2023-01-11T22:03:10.4022907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4023036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4023305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4023443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4023632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8vh98eu
2023-01-11T22:03:10.4023830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8vh98eu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4024091Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4024096Z 
2023-01-11T22:03:10.4024171Z Running tests...
2023-01-11T22:03:10.4024367Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4024656Z   test_backward_rref_nested_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17632
2023-01-11T22:03:10.4024855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17633
2023-01-11T22:03:10.4025045Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17634
2023-01-11T22:03:10.4025202Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17635
2023-01-11T22:03:10.4025480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4025605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4025887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4026023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4026208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdje1hwdu
2023-01-11T22:03:10.4026480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4026594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4026877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4027011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4027195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_x_df3h0
2023-01-11T22:03:10.4027393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdje1hwdu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4027583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_x_df3h0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4027858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4027983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4028253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4028389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4028574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpetctdthx
2023-01-11T22:03:10.4028769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpetctdthx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4029040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4029164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4029446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4029582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4029771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxspi9o_
2023-01-11T22:03:10.4029953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxspi9o_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4030120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4030286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4030452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4030616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4030685Z ok (2.751s)
2023-01-11T22:03:10.4030691Z 
2023-01-11T22:03:10.4030889Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4030966Z Ran 1 test in 2.751s
2023-01-11T22:03:10.4030971Z 
2023-01-11T22:03:10.4031019Z OK
2023-01-11T22:03:10.4031075Z 
2023-01-11T22:03:10.4031146Z Generating XML reports...
2023-01-11T22:03:10.4031607Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215731.xml
2023-01-11T22:03:10.4031885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4032009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4032290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4032427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4032610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5yss10mk
2023-01-11T22:03:10.4032805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5yss10mk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4033050Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4033070Z 
2023-01-11T22:03:10.4033130Z Running tests...
2023-01-11T22:03:10.4033328Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4033612Z   test_backward_rref_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17911
2023-01-11T22:03:10.4033770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17912
2023-01-11T22:03:10.4033926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17913
2023-01-11T22:03:10.4034082Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17914
2023-01-11T22:03:10.4034360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4034484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4034758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4034898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4035085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpde0jw6jr
2023-01-11T22:03:10.4035282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpde0jw6jr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4035559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4035685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4035965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4036102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4036277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpplm8d6xp
2023-01-11T22:03:10.4036476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpplm8d6xp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4036750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4036874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4037158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4037293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4037477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe812nu3f
2023-01-11T22:03:10.4037750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4037874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4038171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4038349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4038539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvf352z5
2023-01-11T22:03:10.4038732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe812nu3f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4039003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvf352z5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4039173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4039341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4039504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4039656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4039725Z ok (2.731s)
2023-01-11T22:03:10.4039730Z 
2023-01-11T22:03:10.4039936Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4040013Z Ran 1 test in 2.731s
2023-01-11T22:03:10.4040019Z 
2023-01-11T22:03:10.4040082Z OK
2023-01-11T22:03:10.4040086Z 
2023-01-11T22:03:10.4040173Z Generating XML reports...
2023-01-11T22:03:10.4040598Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215736.xml
2023-01-11T22:03:10.4040877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4041004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4041270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4041409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4041597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp100ctc7h
2023-01-11T22:03:10.4041791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp100ctc7h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4042049Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4042055Z 
2023-01-11T22:03:10.4042130Z Running tests...
2023-01-11T22:03:10.4042329Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4042622Z   test_backward_simple_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18190
2023-01-11T22:03:10.4042779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18191
2023-01-11T22:03:10.4042926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18192
2023-01-11T22:03:10.4043081Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18193
2023-01-11T22:03:10.4043364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4043491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4043774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4043914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4044105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2rfrkk6
2023-01-11T22:03:10.4044303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2rfrkk6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4044564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4044726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4045038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4045177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4045363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wpk8e66
2023-01-11T22:03:10.4045561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wpk8e66/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4045837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4045964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4046246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4046372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4046560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1yg5wcv
2023-01-11T22:03:10.4046757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1yg5wcv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4047030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4047156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4047438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4047575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4047764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl5oerbnc
2023-01-11T22:03:10.4047950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl5oerbnc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4048119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4048284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4048448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4048610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4048680Z ok (2.847s)
2023-01-11T22:03:10.4048685Z 
2023-01-11T22:03:10.4048885Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4048964Z Ran 1 test in 2.847s
2023-01-11T22:03:10.4048970Z 
2023-01-11T22:03:10.4049032Z OK
2023-01-11T22:03:10.4049037Z 
2023-01-11T22:03:10.4049110Z Generating XML reports...
2023-01-11T22:03:10.4049536Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215741.xml
2023-01-11T22:03:10.4049815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4049940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4050225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4050365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4050552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtlydfix
2023-01-11T22:03:10.4050751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtlydfix/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4051007Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4051012Z 
2023-01-11T22:03:10.4051107Z Running tests...
2023-01-11T22:03:10.4051308Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4051641Z   test_backward_simple_script_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18445
2023-01-11T22:03:10.4051802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18446
2023-01-11T22:03:10.4051958Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18447
2023-01-11T22:03:10.4052112Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18448
2023-01-11T22:03:10.4052393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4052521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4052793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4052933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4053120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25ya0cu7
2023-01-11T22:03:10.4053315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25ya0cu7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4053591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4053716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4053999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4054137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4054321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51jpdrxe
2023-01-11T22:03:10.4054508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51jpdrxe/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4054783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4054908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4055193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4055329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4055514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ezxbxly
2023-01-11T22:03:10.4055710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ezxbxly/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4055984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4056094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4056377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4056515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4056699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfa2qe_fn
2023-01-11T22:03:10.4056894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfa2qe_fn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4057060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4057226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4057390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4057554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4057643Z ok (2.947s)
2023-01-11T22:03:10.4057648Z 
2023-01-11T22:03:10.4057847Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4057989Z Ran 1 test in 2.947s
2023-01-11T22:03:10.4057995Z 
2023-01-11T22:03:10.4058057Z OK
2023-01-11T22:03:10.4058062Z 
2023-01-11T22:03:10.4058149Z Generating XML reports...
2023-01-11T22:03:10.4058577Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215745.xml
2023-01-11T22:03:10.4058856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4058981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4059250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4059388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4059577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuda00j2h
2023-01-11T22:03:10.4059776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuda00j2h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4060033Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4060038Z 
2023-01-11T22:03:10.4060112Z Running tests...
2023-01-11T22:03:10.4060312Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4060599Z   test_backward_simple_self_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18700
2023-01-11T22:03:10.4060758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18701
2023-01-11T22:03:10.4060900Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18702
2023-01-11T22:03:10.4061057Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18703
2023-01-11T22:03:10.4061336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4061462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4061909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4062050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4062238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa1jqhu8n
2023-01-11T22:03:10.4062438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa1jqhu8n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4062716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4062828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4063113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4063251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4063438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprp4e2io1
2023-01-11T22:03:10.4063636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprp4e2io1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4063910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4064035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4064314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4064436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4064685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vccwm6o
2023-01-11T22:03:10.4064909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vccwm6o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4065187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4065311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4065593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4065730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4065916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4c2thdat
2023-01-11T22:03:10.4066111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4c2thdat/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4066265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4066432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4066598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4066765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4066834Z ok (2.640s)
2023-01-11T22:03:10.4066840Z 
2023-01-11T22:03:10.4067040Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4067118Z Ran 1 test in 2.640s
2023-01-11T22:03:10.4067123Z 
2023-01-11T22:03:10.4067186Z OK
2023-01-11T22:03:10.4067191Z 
2023-01-11T22:03:10.4067263Z Generating XML reports...
2023-01-11T22:03:10.4067693Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215750.xml
2023-01-11T22:03:10.4067971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4068099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4068386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4068522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4068709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjolkrk7n
2023-01-11T22:03:10.4068908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjolkrk7n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4069166Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4069172Z 
2023-01-11T22:03:10.4069235Z Running tests...
2023-01-11T22:03:10.4069436Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4069725Z   test_backward_simple_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18955
2023-01-11T22:03:10.4069886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18956
2023-01-11T22:03:10.4070043Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18957
2023-01-11T22:03:10.4070199Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18958
2023-01-11T22:03:10.4070480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4070606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4070889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4071014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4071236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpplwooctt
2023-01-11T22:03:10.4071459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpplwooctt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4071734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4071861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4072144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4072280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4072468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoqk2gkb9
2023-01-11T22:03:10.4072650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoqk2gkb9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4072921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4073047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4073331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4073466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4073649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xl_40vn
2023-01-11T22:03:10.4073843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xl_40vn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4074113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4074237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4074506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4074644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4074832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqguyfsd
2023-01-11T22:03:10.4075029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqguyfsd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4075195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4075358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4075522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4075684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4075740Z ok (2.649s)
2023-01-11T22:03:10.4075759Z 
2023-01-11T22:03:10.4075944Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4076022Z Ran 1 test in 2.650s
2023-01-11T22:03:10.4076028Z 
2023-01-11T22:03:10.4076091Z OK
2023-01-11T22:03:10.4076096Z 
2023-01-11T22:03:10.4076182Z Generating XML reports...
2023-01-11T22:03:10.4076616Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215755.xml
2023-01-11T22:03:10.4076893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4077018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4077303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4077427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4077614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehk50uwu
2023-01-11T22:03:10.4077847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehk50uwu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4078129Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4078135Z 
2023-01-11T22:03:10.4078209Z Running tests...
2023-01-11T22:03:10.4078407Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4078700Z   test_backwards_nested_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19210
2023-01-11T22:03:10.4078926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19211
2023-01-11T22:03:10.4079084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19212
2023-01-11T22:03:10.4079226Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19213
2023-01-11T22:03:10.4079504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4079633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4079922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4080059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4080246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44pz2ai9
2023-01-11T22:03:10.4080442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44pz2ai9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4080715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4080825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4081111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4081248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4081435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_n0n1py
2023-01-11T22:03:10.4081631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_n0n1py/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4081906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4082030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4082311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4082448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4082620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rv6tsvi
2023-01-11T22:03:10.4082818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rv6tsvi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4083091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4083215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4083497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4083631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4083817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn57u1c_i
2023-01-11T22:03:10.4084011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn57u1c_i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4084163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4084327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4084524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4084729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4084800Z ok (2.742s)
2023-01-11T22:03:10.4084805Z 
2023-01-11T22:03:10.4085006Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4085085Z Ran 1 test in 2.742s
2023-01-11T22:03:10.4085091Z 
2023-01-11T22:03:10.4085152Z OK
2023-01-11T22:03:10.4085157Z 
2023-01-11T22:03:10.4085241Z Generating XML reports...
2023-01-11T22:03:10.4085653Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215759.xml
2023-01-11T22:03:10.4085930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4086059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4086342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4086480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4086665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7s56_2c7
2023-01-11T22:03:10.4086858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7s56_2c7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4087118Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4087123Z 
2023-01-11T22:03:10.4087197Z Running tests...
2023-01-11T22:03:10.4087383Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4087677Z   test_context_cleanup_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19465
2023-01-11T22:03:10.4087840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19466
2023-01-11T22:03:10.4087999Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19467
2023-01-11T22:03:10.4088204Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19468
2023-01-11T22:03:10.4088482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4088606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4088892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4089014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4089202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhlc7i67
2023-01-11T22:03:10.4089397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhlc7i67/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4089678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4089805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4090087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4090223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4090408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfj_wp_zg
2023-01-11T22:03:10.4090604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfj_wp_zg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4090862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4090988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4091311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4091477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4091664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mlr2c1u
2023-01-11T22:03:10.4091859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mlr2c1u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4092131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4092256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4092524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4092663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4092850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zju4nb7
2023-01-11T22:03:10.4093046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zju4nb7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4093213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4093377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4093541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4093705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4093884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4094047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4094230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4094406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4094706Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4095010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4095302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4095594Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4095665Z ok (2.842s)
2023-01-11T22:03:10.4095670Z 
2023-01-11T22:03:10.4095864Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4095948Z Ran 1 test in 2.843s
2023-01-11T22:03:10.4095953Z 
2023-01-11T22:03:10.4096012Z OK
2023-01-11T22:03:10.4096018Z 
2023-01-11T22:03:10.4096101Z Generating XML reports...
2023-01-11T22:03:10.4096527Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215804.xml
2023-01-11T22:03:10.4096805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4096917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4097198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4097336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4097523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyal10s8
2023-01-11T22:03:10.4097720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyal10s8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4098016Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4098050Z 
2023-01-11T22:03:10.4098124Z Running tests...
2023-01-11T22:03:10.4098321Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4098619Z   test_context_cleanup_tensor_no_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19712
2023-01-11T22:03:10.4098767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19713
2023-01-11T22:03:10.4098925Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19714
2023-01-11T22:03:10.4099080Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19715
2023-01-11T22:03:10.4099357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4099487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4099773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4099910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4100099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprk4r_nfs
2023-01-11T22:03:10.4100281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprk4r_nfs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4100554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4100678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4100960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4101098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4101287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcspj6rs
2023-01-11T22:03:10.4101485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcspj6rs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4101901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4102027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4102296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4102432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4102615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ch3j0ow
2023-01-11T22:03:10.4102808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ch3j0ow/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4103086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4103210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4103496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4103632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4103841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfaq84s5z
2023-01-11T22:03:10.4104039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfaq84s5z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4104204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4104368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4104588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4104782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4104963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4105141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4105316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4105603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4105904Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4106082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4106382Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4106684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4106754Z ok (2.946s)
2023-01-11T22:03:10.4106760Z 
2023-01-11T22:03:10.4106959Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4107036Z Ran 1 test in 2.946s
2023-01-11T22:03:10.4107042Z 
2023-01-11T22:03:10.4107107Z OK
2023-01-11T22:03:10.4107112Z 
2023-01-11T22:03:10.4107185Z Generating XML reports...
2023-01-11T22:03:10.4107617Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215809.xml
2023-01-11T22:03:10.4107896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4108022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4108307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4108441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4108626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6whmtoto
2023-01-11T22:03:10.4108826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6whmtoto/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4109082Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4109087Z 
2023-01-11T22:03:10.4109150Z Running tests...
2023-01-11T22:03:10.4109345Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4109643Z   test_context_cleanup_tensor_with_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19959
2023-01-11T22:03:10.4109807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19960
2023-01-11T22:03:10.4109961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19961
2023-01-11T22:03:10.4110114Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19962
2023-01-11T22:03:10.4110392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4110515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4110787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4110921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4111103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdort94v6
2023-01-11T22:03:10.4111333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdort94v6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4111634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4111758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4112038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4112171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4112354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0dutw2i1
2023-01-11T22:03:10.4112536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0dutw2i1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4112805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4112930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4113212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4113347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4113530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaetytyzs
2023-01-11T22:03:10.4113725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaetytyzs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4113995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4114117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4114381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4114517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4114701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoz5pzjwr
2023-01-11T22:03:10.4114897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoz5pzjwr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4115060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4115224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4115385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4115546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4115709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4115883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4116059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4116361Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4116532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4116829Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4117126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4117418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4117488Z ok (2.842s)
2023-01-11T22:03:10.4117494Z 
2023-01-11T22:03:10.4117717Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4117792Z Ran 1 test in 2.842s
2023-01-11T22:03:10.4117798Z 
2023-01-11T22:03:10.4117859Z OK
2023-01-11T22:03:10.4117865Z 
2023-01-11T22:03:10.4117982Z Generating XML reports...
2023-01-11T22:03:10.4118409Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215814.xml
2023-01-11T22:03:10.4118681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4118805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4119183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4119320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4119493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5c0ilt2u
2023-01-11T22:03:10.4119691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5c0ilt2u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4119948Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4119953Z 
2023-01-11T22:03:10.4120027Z Running tests...
2023-01-11T22:03:10.4120229Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4120523Z   test_embedding_bag_with_no_grad_tensors (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20206
2023-01-11T22:03:10.4120683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20207
2023-01-11T22:03:10.4120839Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20208
2023-01-11T22:03:10.4120994Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20209
2023-01-11T22:03:10.4121259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4121385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4121668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4121806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4121992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4a2czsfe
2023-01-11T22:03:10.4122187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4a2czsfe/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4122460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4122583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4122852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4122988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4123173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rfveo7g
2023-01-11T22:03:10.4123369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rfveo7g/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4123641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4123766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4124044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4124178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4124361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8ifd7qc
2023-01-11T22:03:10.4124581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8ifd7qc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4124882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4125007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4125288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4125421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4125603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7u2p0aqm
2023-01-11T22:03:10.4125798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7u2p0aqm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4125962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4126117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4126281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4126442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4126509Z ok (2.752s)
2023-01-11T22:03:10.4126515Z 
2023-01-11T22:03:10.4126709Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4126785Z Ran 1 test in 2.752s
2023-01-11T22:03:10.4126791Z 
2023-01-11T22:03:10.4126852Z OK
2023-01-11T22:03:10.4126857Z 
2023-01-11T22:03:10.4126939Z Generating XML reports...
2023-01-11T22:03:10.4127364Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215819.xml
2023-01-11T22:03:10.4127627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4127753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4128035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4128172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4128353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6nbco33a
2023-01-11T22:03:10.4128546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6nbco33a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4128802Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4128807Z 
2023-01-11T22:03:10.4128879Z Running tests...
2023-01-11T22:03:10.4129062Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4129347Z   test_graph_for_builtin_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20473
2023-01-11T22:03:10.4129505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20474
2023-01-11T22:03:10.4129660Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20475
2023-01-11T22:03:10.4129813Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20476
2023-01-11T22:03:10.4130091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4130214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4130496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4130631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4130804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0akxkphe
2023-01-11T22:03:10.4131032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0akxkphe/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4131346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4131471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4131755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4131889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4132072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptp_btei1
2023-01-11T22:03:10.4132266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptp_btei1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4132539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4132652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4132934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4133067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4133251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7vv85z9x
2023-01-11T22:03:10.4133442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7vv85z9x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4133713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4133837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4134116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4134239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4134425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8yepdkm
2023-01-11T22:03:10.4134624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8yepdkm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4134789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4134953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4135116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4135278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4135455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4135634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4135799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4135975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4136277Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4136578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4136866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4137156Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4137226Z ok (2.849s)
2023-01-11T22:03:10.4137232Z 
2023-01-11T22:03:10.4137430Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4137557Z Ran 1 test in 2.849s
2023-01-11T22:03:10.4137564Z 
2023-01-11T22:03:10.4137613Z OK
2023-01-11T22:03:10.4137618Z 
2023-01-11T22:03:10.4137727Z Generating XML reports...
2023-01-11T22:03:10.4138152Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215823.xml
2023-01-11T22:03:10.4138429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4138554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4138835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4138970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4139156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0x3d9n8
2023-01-11T22:03:10.4139352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0x3d9n8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4139597Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4139603Z 
2023-01-11T22:03:10.4139678Z Running tests...
2023-01-11T22:03:10.4139874Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4140168Z   test_graph_for_builtin_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20720
2023-01-11T22:03:10.4140327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20721
2023-01-11T22:03:10.4140483Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20722
2023-01-11T22:03:10.4140641Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20723
2023-01-11T22:03:10.4140919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4141032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4141317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4141454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4141639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6k952ur
2023-01-11T22:03:10.4142010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6k952ur/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4142290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4142415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4142698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4142838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4143010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsmbc_213
2023-01-11T22:03:10.4143203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsmbc_213/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4143474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4143597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4143878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4144014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4144198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxaj3buh
2023-01-11T22:03:10.4144445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxaj3buh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4144751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4144863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4145143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4145278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4145464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzn3pn8f3
2023-01-11T22:03:10.4145658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzn3pn8f3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4145824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4145987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4146154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4146306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4146481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4146658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4146957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4147127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4147297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4147592Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4147892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4148181Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4148236Z ok (2.949s)
2023-01-11T22:03:10.4148256Z 
2023-01-11T22:03:10.4148440Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4148516Z Ran 1 test in 2.949s
2023-01-11T22:03:10.4148522Z 
2023-01-11T22:03:10.4148584Z OK
2023-01-11T22:03:10.4148589Z 
2023-01-11T22:03:10.4148674Z Generating XML reports...
2023-01-11T22:03:10.4149100Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215828.xml
2023-01-11T22:03:10.4149374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4149501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4149784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4149906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4150092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4t9js9l1
2023-01-11T22:03:10.4150286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4t9js9l1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4150537Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4150543Z 
2023-01-11T22:03:10.4150615Z Running tests...
2023-01-11T22:03:10.4150811Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4151107Z   test_graph_for_py_nested_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20967
2023-01-11T22:03:10.4151325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20968
2023-01-11T22:03:10.4151480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20969
2023-01-11T22:03:10.4151621Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20970
2023-01-11T22:03:10.4151899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4152023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4152307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4152443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4152632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7l0ette
2023-01-11T22:03:10.4152826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7l0ette/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4153104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4153214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4153495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4153630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4153814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ua1yfdg
2023-01-11T22:03:10.4154011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ua1yfdg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4154281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4154409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4154805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4154942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4155114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpauvm9a18
2023-01-11T22:03:10.4155309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpauvm9a18/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4155583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4155707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4155989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4156128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4156317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7jnr6nb
2023-01-11T22:03:10.4156511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7jnr6nb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4156664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4156831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4156996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4157160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4157339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4157555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4157754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4157933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4158235Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4158521Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4158812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4159204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4159275Z ok (2.850s)
2023-01-11T22:03:10.4159284Z 
2023-01-11T22:03:10.4159485Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4159562Z Ran 1 test in 2.850s
2023-01-11T22:03:10.4159567Z 
2023-01-11T22:03:10.4159631Z OK
2023-01-11T22:03:10.4159636Z 
2023-01-11T22:03:10.4159721Z Generating XML reports...
2023-01-11T22:03:10.4160150Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215833.xml
2023-01-11T22:03:10.4160414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4160541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4160821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4160958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4161146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7utk5pp
2023-01-11T22:03:10.4161347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7utk5pp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4161602Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4161607Z 
2023-01-11T22:03:10.4161683Z Running tests...
2023-01-11T22:03:10.4161882Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4162160Z   test_graph_for_py_nested_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21214
2023-01-11T22:03:10.4162321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21215
2023-01-11T22:03:10.4162474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21216
2023-01-11T22:03:10.4162627Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21217
2023-01-11T22:03:10.4162910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4163039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4163327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4163465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4163639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpse6bx5d7
2023-01-11T22:03:10.4163842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpse6bx5d7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4164119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4164247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4164603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4164766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4164953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7k9bn4_7
2023-01-11T22:03:10.4165147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7k9bn4_7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4165553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4165713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4165997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4166131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4166314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqkrgccr
2023-01-11T22:03:10.4166511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqkrgccr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4166782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4166905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4167184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4167317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4167487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprp92_hku
2023-01-11T22:03:10.4167678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprp92_hku/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4167843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4168009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4168172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4168332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4168514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4168691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4168850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4169151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4169320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4169614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4169913Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4170203Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4170270Z ok (2.949s)
2023-01-11T22:03:10.4170277Z 
2023-01-11T22:03:10.4170473Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4170548Z Ran 1 test in 2.950s
2023-01-11T22:03:10.4170554Z 
2023-01-11T22:03:10.4170602Z OK
2023-01-11T22:03:10.4170618Z 
2023-01-11T22:03:10.4170690Z Generating XML reports...
2023-01-11T22:03:10.4171118Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215838.xml
2023-01-11T22:03:10.4171444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4171601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4171885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4172022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4172209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcbrzpz_
2023-01-11T22:03:10.4172406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcbrzpz_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4172649Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4172654Z 
2023-01-11T22:03:10.4172726Z Running tests...
2023-01-11T22:03:10.4172923Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4173233Z   test_graph_for_py_nested_remote_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21461
2023-01-11T22:03:10.4173393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21462
2023-01-11T22:03:10.4173551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21463
2023-01-11T22:03:10.4173705Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21464
2023-01-11T22:03:10.4173985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4174111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4174380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4174515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4174704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpga43ieup
2023-01-11T22:03:10.4174901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpga43ieup/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4175173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4175296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4175575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4175708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4175879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk48mcfhd
2023-01-11T22:03:10.4176080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk48mcfhd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4176351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4176477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4176754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4176888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4177069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tun09we
2023-01-11T22:03:10.4177338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4177459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4177725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4177891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4178075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqm4seqyr
2023-01-11T22:03:10.4178315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tun09we/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4178504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqm4seqyr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4178668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4178831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4178993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4179145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4179320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4179498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4179800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4180099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4180273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4180447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4180741Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4181036Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4181094Z ok (2.838s)
2023-01-11T22:03:10.4181112Z 
2023-01-11T22:03:10.4181299Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4181378Z Ran 1 test in 2.838s
2023-01-11T22:03:10.4181383Z 
2023-01-11T22:03:10.4181448Z OK
2023-01-11T22:03:10.4181452Z 
2023-01-11T22:03:10.4181538Z Generating XML reports...
2023-01-11T22:03:10.4182121Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215843.xml
2023-01-11T22:03:10.4182399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4182525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4182807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4182929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4183120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2q46mxir
2023-01-11T22:03:10.4183318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2q46mxir/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4183575Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4183581Z 
2023-01-11T22:03:10.4183659Z Running tests...
2023-01-11T22:03:10.4183857Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4184158Z   test_graph_for_py_nested_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21708
2023-01-11T22:03:10.4184319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21709
2023-01-11T22:03:10.4184478Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21710
2023-01-11T22:03:10.4184681Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21711
2023-01-11T22:03:10.4184995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4185123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4185408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4185545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4185733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7majfhso
2023-01-11T22:03:10.4185932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7majfhso/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4186207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4186339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4186608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4186745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4186932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7f1pbr3
2023-01-11T22:03:10.4187128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7f1pbr3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4187402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4187527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4187807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4187943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4188118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu1vk3iyo
2023-01-11T22:03:10.4188314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu1vk3iyo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4188583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4188705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4188985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4189120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4189305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpf887kr
2023-01-11T22:03:10.4189498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpf887kr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4189665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4189819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4189983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4190149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4190327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4190502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4190676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4190847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4191146Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4191507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4191792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4192088Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4192154Z ok (2.849s)
2023-01-11T22:03:10.4192160Z 
2023-01-11T22:03:10.4192358Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4192431Z Ran 1 test in 2.849s
2023-01-11T22:03:10.4192437Z 
2023-01-11T22:03:10.4192496Z OK
2023-01-11T22:03:10.4192501Z 
2023-01-11T22:03:10.4192584Z Generating XML reports...
2023-01-11T22:03:10.4193006Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215847.xml
2023-01-11T22:03:10.4193277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4193401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4193680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4193814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4194002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytndhfup
2023-01-11T22:03:10.4194199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytndhfup/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4194452Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4194457Z 
2023-01-11T22:03:10.4194532Z Running tests...
2023-01-11T22:03:10.4194730Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4195004Z   test_graph_for_python_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21955
2023-01-11T22:03:10.4195162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21956
2023-01-11T22:03:10.4195315Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21957
2023-01-11T22:03:10.4195468Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21958
2023-01-11T22:03:10.4195744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4195866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4196149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4196286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4196473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5q625nt
2023-01-11T22:03:10.4196653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5q625nt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4196923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4197046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4197326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4197459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4197643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9skx4fms
2023-01-11T22:03:10.4197912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4198066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4198359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4198493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4198676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwhp8780o
2023-01-11T22:03:10.4198942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9skx4fms/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4199220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4199346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4199624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4199763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4199950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrgstl_i
2023-01-11T22:03:10.4200131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwhp8780o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4200326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrgstl_i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4200493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4200660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4200823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4200985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4201166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4201346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4201508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4201809Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4201984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4202283Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4202583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4202874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4202947Z ok (2.946s)
2023-01-11T22:03:10.4202954Z 
2023-01-11T22:03:10.4203154Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4203230Z Ran 1 test in 2.946s
2023-01-11T22:03:10.4203236Z 
2023-01-11T22:03:10.4203286Z OK
2023-01-11T22:03:10.4203302Z 
2023-01-11T22:03:10.4203373Z Generating XML reports...
2023-01-11T22:03:10.4203801Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215852.xml
2023-01-11T22:03:10.4204075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4204202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4204487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4204658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4204875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpym6uul_d
2023-01-11T22:03:10.4205074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpym6uul_d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4205316Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4205333Z 
2023-01-11T22:03:10.4205393Z Running tests...
2023-01-11T22:03:10.4205591Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4205884Z   test_graph_for_python_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22202
2023-01-11T22:03:10.4206040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22203
2023-01-11T22:03:10.4206196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22204
2023-01-11T22:03:10.4206349Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22205
2023-01-11T22:03:10.4206627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4206750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4207020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4207155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4207341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wjl4pu5
2023-01-11T22:03:10.4207536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wjl4pu5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4207811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4207937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4208220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4208356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4208527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46e3v_5l
2023-01-11T22:03:10.4208720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46e3v_5l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4208994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4209116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4209398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4209534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4209720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjusgy_7l
2023-01-11T22:03:10.4209913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjusgy_7l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4210187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4210298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4210576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4210710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4210894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7zjdgku
2023-01-11T22:03:10.4211089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7zjdgku/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4211283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4211478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4211641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4211803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4211971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4212147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4212323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4212496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4212802Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4213102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4213398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4213698Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4213764Z ok (2.849s)
2023-01-11T22:03:10.4213769Z 
2023-01-11T22:03:10.4213954Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4214032Z Ran 1 test in 2.849s
2023-01-11T22:03:10.4214037Z 
2023-01-11T22:03:10.4214100Z OK
2023-01-11T22:03:10.4214105Z 
2023-01-11T22:03:10.4214190Z Generating XML reports...
2023-01-11T22:03:10.4214617Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215857.xml
2023-01-11T22:03:10.4214897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4215023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4215308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4215431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4215622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvqy2i40
2023-01-11T22:03:10.4215819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvqy2i40/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4216074Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4216082Z 
2023-01-11T22:03:10.4216161Z Running tests...
2023-01-11T22:03:10.4216359Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4216649Z   test_mixed_requires_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22449
2023-01-11T22:03:10.4216811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22450
2023-01-11T22:03:10.4216968Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22451
2023-01-11T22:03:10.4217109Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22452
2023-01-11T22:03:10.4217390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4217515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4217797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4217962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4218176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03zyr94u
2023-01-11T22:03:10.4218373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03zyr94u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4218648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4218773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4219043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4219179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4219364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrr3n9q6
2023-01-11T22:03:10.4219562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrr3n9q6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4219837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4219962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4220242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4220378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4220549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsikernsf
2023-01-11T22:03:10.4220742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsikernsf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4221012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4221140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4221427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4221563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4221917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpre3yv5b7
2023-01-11T22:03:10.4222125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpre3yv5b7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4222289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4222441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4222604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4222766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4222836Z ok (2.742s)
2023-01-11T22:03:10.4222843Z 
2023-01-11T22:03:10.4223046Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4223122Z Ran 1 test in 2.742s
2023-01-11T22:03:10.4223127Z 
2023-01-11T22:03:10.4223189Z OK
2023-01-11T22:03:10.4223194Z 
2023-01-11T22:03:10.4223279Z Generating XML reports...
2023-01-11T22:03:10.4223697Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215902.xml
2023-01-11T22:03:10.4223971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4224095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4224377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4224571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4224757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9tbii84
2023-01-11T22:03:10.4225001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9tbii84/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4225256Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4225261Z 
2023-01-11T22:03:10.4225336Z Running tests...
2023-01-11T22:03:10.4225522Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4225806Z   test_multiple_backward_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22704
2023-01-11T22:03:10.4225964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22705
2023-01-11T22:03:10.4226118Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22706
2023-01-11T22:03:10.4226275Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22707
2023-01-11T22:03:10.4226553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4226675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4226958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4227093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4227264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwu2k7_to
2023-01-11T22:03:10.4227459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwu2k7_to/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4227736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4227862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4228144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4228280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4228463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97qbggyh
2023-01-11T22:03:10.4228656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97qbggyh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4228913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4229036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4229316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4229448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4229634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqo6tonb2
2023-01-11T22:03:10.4229828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqo6tonb2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4230098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4230222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4230500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4230621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4230804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3aogt91
2023-01-11T22:03:10.4230995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3aogt91/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4231194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4231385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4231548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4231709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4231775Z ok (3.743s)
2023-01-11T22:03:10.4231781Z 
2023-01-11T22:03:10.4231967Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4232042Z Ran 1 test in 3.743s
2023-01-11T22:03:10.4232048Z 
2023-01-11T22:03:10.4232109Z OK
2023-01-11T22:03:10.4232114Z 
2023-01-11T22:03:10.4232198Z Generating XML reports...
2023-01-11T22:03:10.4232624Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215907.xml
2023-01-11T22:03:10.4232898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4233024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4233303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4233437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4233607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7o2rqjmb
2023-01-11T22:03:10.4233799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7o2rqjmb/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4234059Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4234065Z 
2023-01-11T22:03:10.4234141Z Running tests...
2023-01-11T22:03:10.4234337Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4234642Z   test_nested_backward_accumulate_grads_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22959
2023-01-11T22:03:10.4234800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22960
2023-01-11T22:03:10.4234955Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22961
2023-01-11T22:03:10.4235107Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22962
2023-01-11T22:03:10.4235370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4235493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4235776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4235912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4236100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpln2uk2_k
2023-01-11T22:03:10.4236299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpln2uk2_k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4236573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4236697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4236964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4237103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4237288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf30zdpic
2023-01-11T22:03:10.4237483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf30zdpic/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4237786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4237939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4238221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4238354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4238539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqfx7zhpo
2023-01-11T22:03:10.4238722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqfx7zhpo/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4239068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4239195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4239480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4239616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4239803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphg3dx7x7
2023-01-11T22:03:10.4239999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphg3dx7x7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4240164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4240315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4240477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4240637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4240706Z ok (2.750s)
2023-01-11T22:03:10.4240715Z 
2023-01-11T22:03:10.4240918Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4240993Z Ran 1 test in 2.750s
2023-01-11T22:03:10.4240999Z 
2023-01-11T22:03:10.4241060Z OK
2023-01-11T22:03:10.4241068Z 
2023-01-11T22:03:10.4241152Z Generating XML reports...
2023-01-11T22:03:10.4241577Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215912.xml
2023-01-11T22:03:10.4241839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4241963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4242242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4242377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4242563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyn1ipi56
2023-01-11T22:03:10.4242758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyn1ipi56/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4243014Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4243020Z 
2023-01-11T22:03:10.4243092Z Running tests...
2023-01-11T22:03:10.4243277Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4243586Z   test_no_graph_with_tensors_not_require_grad_remote_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23214
2023-01-11T22:03:10.4243743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23215
2023-01-11T22:03:10.4243897Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23216
2023-01-11T22:03:10.4244050Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23217
2023-01-11T22:03:10.4244373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4244529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4244812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4244948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4245120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3af7g1gc
2023-01-11T22:03:10.4245314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3af7g1gc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4245587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4245711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4245993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4246128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4246310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr8fsy0wi
2023-01-11T22:03:10.4246502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr8fsy0wi/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4246770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4246881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4247158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4247294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4247478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo031ffz3
2023-01-11T22:03:10.4247674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo031ffz3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4247948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4248072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4248350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4248470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4248651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9i2008b
2023-01-11T22:03:10.4248840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9i2008b/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4249004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4249170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4249332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4249495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4249671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4249848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4250011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4250313Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4250485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4250816Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4251140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4251432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4251501Z ok (2.848s)
2023-01-11T22:03:10.4251507Z 
2023-01-11T22:03:10.4251706Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4251780Z Ran 1 test in 2.848s
2023-01-11T22:03:10.4251786Z 
2023-01-11T22:03:10.4251834Z OK
2023-01-11T22:03:10.4251838Z 
2023-01-11T22:03:10.4251921Z Generating XML reports...
2023-01-11T22:03:10.4252350Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215917.xml
2023-01-11T22:03:10.4252628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4252756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4253040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4253176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4253361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7gbn5_oy
2023-01-11T22:03:10.4253556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7gbn5_oy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4253798Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4253803Z 
2023-01-11T22:03:10.4253876Z Running tests...
2023-01-11T22:03:10.4254071Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4254378Z   test_no_graph_with_tensors_not_require_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23461
2023-01-11T22:03:10.4254535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23462
2023-01-11T22:03:10.4254690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23463
2023-01-11T22:03:10.4254842Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23464
2023-01-11T22:03:10.4255116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4255227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4255511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4255644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4255830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpml3m_ffd
2023-01-11T22:03:10.4256027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpml3m_ffd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4256297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4256420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4256699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4256833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4257004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc75fsrb8
2023-01-11T22:03:10.4257196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc75fsrb8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4257499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4257649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4257933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4258069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4258255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazymqj_f
2023-01-11T22:03:10.4258453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazymqj_f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4258725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4258836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4259118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4259252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4259440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppp6u8c21
2023-01-11T22:03:10.4259633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppp6u8c21/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4259796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4259959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4260120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4260269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4260446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4260625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4260804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4260977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4261278Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4261581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4261980Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4262275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4262333Z ok (2.945s)
2023-01-11T22:03:10.4262351Z 
2023-01-11T22:03:10.4262539Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4262618Z Ran 1 test in 2.945s
2023-01-11T22:03:10.4262624Z 
2023-01-11T22:03:10.4262685Z OK
2023-01-11T22:03:10.4262690Z 
2023-01-11T22:03:10.4262776Z Generating XML reports...
2023-01-11T22:03:10.4263204Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215922.xml
2023-01-11T22:03:10.4263481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4263610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4263893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4264016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4264256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9pvpsk5i
2023-01-11T22:03:10.4264521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9pvpsk5i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4264778Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4264784Z 
2023-01-11T22:03:10.4264858Z Running tests...
2023-01-11T22:03:10.4265053Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4265342Z   test_remote_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23708
2023-01-11T22:03:10.4265500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23709
2023-01-11T22:03:10.4265656Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23710
2023-01-11T22:03:10.4265800Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23711
2023-01-11T22:03:10.4266080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4266206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4266488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4266624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4266812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4mhw71r
2023-01-11T22:03:10.4267009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4mhw71r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4267282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4267395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4267679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4267821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4268005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqe4e55r
2023-01-11T22:03:10.4268200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqe4e55r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4268472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4268597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4268877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4269012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4269185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_9zlu2r
2023-01-11T22:03:10.4269382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_9zlu2r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4269652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4269777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4270055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4270191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4270375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzb22f753
2023-01-11T22:03:10.4270568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzb22f753/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4270749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4270957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4271122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4271284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4271353Z ok (2.646s)
2023-01-11T22:03:10.4271358Z 
2023-01-11T22:03:10.4271557Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4271635Z Ran 1 test in 2.646s
2023-01-11T22:03:10.4271640Z 
2023-01-11T22:03:10.4271703Z OK
2023-01-11T22:03:10.4271708Z 
2023-01-11T22:03:10.4271794Z Generating XML reports...
2023-01-11T22:03:10.4272210Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215927.xml
2023-01-11T22:03:10.4272490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4272616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4272899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4273038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4273224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9o1vc5u7
2023-01-11T22:03:10.4273421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9o1vc5u7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4273678Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4273684Z 
2023-01-11T22:03:10.4273759Z Running tests...
2023-01-11T22:03:10.4273944Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4274231Z   test_rpc_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23955
2023-01-11T22:03:10.4274393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23956
2023-01-11T22:03:10.4274551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23957
2023-01-11T22:03:10.4274706Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23958
2023-01-11T22:03:10.4274986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4275112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4275399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4275522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4275714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8_tbuzr
2023-01-11T22:03:10.4275914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8_tbuzr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4276187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4276310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4276593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4276730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4276917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3h7ysb61
2023-01-11T22:03:10.4277113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3h7ysb61/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4277407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4277536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4277850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4277984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4278171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqc7hwqa_
2023-01-11T22:03:10.4278440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4278564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4278912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4279037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4279225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9qbs5dg
2023-01-11T22:03:10.4279422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqc7hwqa_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4279614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9qbs5dg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4279781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4279946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4280108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4280270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4280337Z ok (2.852s)
2023-01-11T22:03:10.4280342Z 
2023-01-11T22:03:10.4280529Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4280608Z Ran 1 test in 2.852s
2023-01-11T22:03:10.4280613Z 
2023-01-11T22:03:10.4280676Z OK
2023-01-11T22:03:10.4280681Z 
2023-01-11T22:03:10.4280768Z Generating XML reports...
2023-01-11T22:03:10.4281197Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215931.xml
2023-01-11T22:03:10.4281470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4281595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4281879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4282002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4282188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ay57lub
2023-01-11T22:03:10.4282382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ay57lub/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4282636Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4282641Z 
2023-01-11T22:03:10.4282714Z Running tests...
2023-01-11T22:03:10.4282911Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4283188Z   test_trainer_ps_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24202
2023-01-11T22:03:10.4283344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24203
2023-01-11T22:03:10.4283499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24204
2023-01-11T22:03:10.4283640Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24205
2023-01-11T22:03:10.4283913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4284072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4284379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4284518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4284708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1yc8sw_
2023-01-11T22:03:10.4284903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1yc8sw_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4285175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4285297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4285564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4285702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4285888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5be7l50
2023-01-11T22:03:10.4286082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5be7l50/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4286353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4286478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4286760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4286894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4287065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnx0kqcc
2023-01-11T22:03:10.4287261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnx0kqcc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4287534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4287659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4287939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4288075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4288259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7suikew
2023-01-11T22:03:10.4288451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7suikew/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4288615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4288766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4288929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4289094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4289161Z ok (2.966s)
2023-01-11T22:03:10.4289167Z 
2023-01-11T22:03:10.4289365Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4289443Z Ran 1 test in 2.966s
2023-01-11T22:03:10.4289448Z 
2023-01-11T22:03:10.4289509Z OK
2023-01-11T22:03:10.4289515Z 
2023-01-11T22:03:10.4289598Z Generating XML reports...
2023-01-11T22:03:10.4290006Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215936.xml
2023-01-11T22:03:10.4290281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4290443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4290724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4290886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4291072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp82njat4
2023-01-11T22:03:10.4291269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp82njat4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4291525Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4291531Z 
2023-01-11T22:03:10.4291604Z Running tests...
2023-01-11T22:03:10.4291788Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4292051Z   test_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24469
2023-01-11T22:03:10.4292211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24470
2023-01-11T22:03:10.4292365Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24471
2023-01-11T22:03:10.4292519Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24472
2023-01-11T22:03:10.4292793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4292915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4293194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4293316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4293500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_xxuoqw
2023-01-11T22:03:10.4293694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_xxuoqw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4293968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4294092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4294372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4294507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4294691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjx8tw30z
2023-01-11T22:03:10.4294885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjx8tw30z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4295142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4295265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4295544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4295681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4295865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgf4c587n
2023-01-11T22:03:10.4296057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgf4c587n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4296329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4296453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4296732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4296854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4297074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm492_ctu
2023-01-11T22:03:10.4297293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm492_ctu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4297457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4297621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4297782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4297942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4298010Z ok (2.746s)
2023-01-11T22:03:10.4298016Z 
2023-01-11T22:03:10.4298202Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4298277Z Ran 1 test in 2.746s
2023-01-11T22:03:10.4298282Z 
2023-01-11T22:03:10.4298342Z OK
2023-01-11T22:03:10.4298350Z 
2023-01-11T22:03:10.4303191Z Generating XML reports...
2023-01-11T22:03:10.4303672Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111215941.xml
2023-01-11T22:03:10.4303959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4304093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4304382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4304521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4304711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1p2t_lyf
2023-01-11T22:03:10.4304912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1p2t_lyf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4305170Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4305180Z 
2023-01-11T22:03:10.4305257Z Running tests...
2023-01-11T22:03:10.4305449Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4305716Z   test_builtin_remote_self_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24704
2023-01-11T22:03:10.4305875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24705
2023-01-11T22:03:10.4306031Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24706
2023-01-11T22:03:10.4306186Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24707
2023-01-11T22:03:10.4306465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4306592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4306880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4307007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4307197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2c9uwfa5
2023-01-11T22:03:10.4307397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2c9uwfa5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4307672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4307798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4308081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4308219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4308513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7anhy5v
2023-01-11T22:03:10.4308710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7anhy5v/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4309011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4309139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4309424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4309562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4309748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtlpjxhn
2023-01-11T22:03:10.4309949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtlpjxhn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4310223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4310351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4310620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4310759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4310944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjg5os5c3
2023-01-11T22:03:10.4311141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjg5os5c3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4311306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4311473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4311637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4311802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4311871Z ok (2.649s)
2023-01-11T22:03:10.4311877Z 
2023-01-11T22:03:10.4312068Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4312145Z Ran 1 test in 2.650s
2023-01-11T22:03:10.4312151Z 
2023-01-11T22:03:10.4312214Z OK
2023-01-11T22:03:10.4312220Z 
2023-01-11T22:03:10.4312305Z Generating XML reports...
2023-01-11T22:03:10.4312704Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111215945.xml
2023-01-11T22:03:10.4312989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4313115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4313396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4313521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4313714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpji525f99
2023-01-11T22:03:10.4313910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpji525f99/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4314171Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4314177Z 
2023-01-11T22:03:10.4314252Z Running tests...
2023-01-11T22:03:10.4314451Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4314727Z   test_dynamic_and_static_init_rpc_together (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24939
2023-01-11T22:03:10.4314887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24940
2023-01-11T22:03:10.4315084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24941
2023-01-11T22:03:10.4315226Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24942
2023-01-11T22:03:10.4315533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4315663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4315945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4316081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4316268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkzp4091i
2023-01-11T22:03:10.4316464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkzp4091i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4316738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4316866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4317136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4317272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4317460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01sd0yv5
2023-01-11T22:03:10.4317656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01sd0yv5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4317931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4318057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4318340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4318481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4318654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3yvljjt_
2023-01-11T22:03:10.4318920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3yvljjt_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4319200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4319327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4319609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4319746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4319933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkglrrxn
2023-01-11T22:03:10.4320131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkglrrxn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4320299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4320454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4320619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4320781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4320961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4321141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4321440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4321615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4321824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4322163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4322454Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4322746Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4323021Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4323296Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4323371Z ok (2.840s)
2023-01-11T22:03:10.4323377Z 
2023-01-11T22:03:10.4323581Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4323659Z Ran 1 test in 2.841s
2023-01-11T22:03:10.4323665Z 
2023-01-11T22:03:10.4323727Z OK
2023-01-11T22:03:10.4323732Z 
2023-01-11T22:03:10.4323817Z Generating XML reports...
2023-01-11T22:03:10.4324200Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111215950.xml
2023-01-11T22:03:10.4324480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4324611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4324898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4325039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4325231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3z1p5yn
2023-01-11T22:03:10.4325432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3z1p5yn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4325692Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4325698Z 
2023-01-11T22:03:10.4325774Z Running tests...
2023-01-11T22:03:10.4325960Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4326745Z   test_dynamic_rpc_existing_rank_can_communicate_with_new_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84116 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.328s)
2023-01-11T22:03:10.4326767Z 
2023-01-11T22:03:10.4326951Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4327031Z Ran 1 test in 0.328s
2023-01-11T22:03:10.4327037Z 
2023-01-11T22:03:10.4327111Z OK (skipped=1)
2023-01-11T22:03:10.4327116Z 
2023-01-11T22:03:10.4327203Z Generating XML reports...
2023-01-11T22:03:10.4327596Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111215955.xml
2023-01-11T22:03:10.4327875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4328002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4328290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4328413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4328637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpel0d1gdx
2023-01-11T22:03:10.4328865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpel0d1gdx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4329122Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4329128Z 
2023-01-11T22:03:10.4329203Z Running tests...
2023-01-11T22:03:10.4329401Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4329699Z   test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25159
2023-01-11T22:03:10.4329859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25160
2023-01-11T22:03:10.4330015Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25161
2023-01-11T22:03:10.4330161Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25162
2023-01-11T22:03:10.4330439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4330566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4330852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4330989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4331175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdyladja
2023-01-11T22:03:10.4331376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdyladja/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4331650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4331764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4332050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4332188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4332372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplw5kiiz7
2023-01-11T22:03:10.4332567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplw5kiiz7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4332840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4332967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4333249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4333385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4333560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwg7gc0i
2023-01-11T22:03:10.4333839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4333963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4334242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4334379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4334563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgs_ivct
2023-01-11T22:03:10.4334760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwg7gc0i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4334951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgs_ivct/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4335149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4335331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4335497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4335664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4335771Z skip: Need at least 2 CUDA devices (2.343s)
2023-01-11T22:03:10.4335777Z 
2023-01-11T22:03:10.4335974Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4336051Z Ran 1 test in 2.344s
2023-01-11T22:03:10.4336056Z 
2023-01-11T22:03:10.4336130Z OK (skipped=1)
2023-01-11T22:03:10.4336135Z 
2023-01-11T22:03:10.4336220Z Generating XML reports...
2023-01-11T22:03:10.4336604Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111215957.xml
2023-01-11T22:03:10.4336884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4337011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4337291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4337428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4337615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhhgtbjj
2023-01-11T22:03:10.4337816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhhgtbjj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4338071Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4338077Z 
2023-01-11T22:03:10.4338151Z Running tests...
2023-01-11T22:03:10.4338338Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4338598Z   test_dynamic_rpc_init_rpc (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25214
2023-01-11T22:03:10.4338758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25215
2023-01-11T22:03:10.4338916Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25216
2023-01-11T22:03:10.4339070Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25217
2023-01-11T22:03:10.4339346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4339471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4339757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4339880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4340072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp99jdid2
2023-01-11T22:03:10.4340272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp99jdid2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4340551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4340678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4340961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4341099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4341285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0tt92sc7
2023-01-11T22:03:10.4341561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4341809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4342144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4342283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4342468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpav9aai4q
2023-01-11T22:03:10.4342666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0tt92sc7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4342860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpav9aai4q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4343138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4343264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4343546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4343671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4343861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsri85ui0
2023-01-11T22:03:10.4344056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsri85ui0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4344222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4344386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4344552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4344716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4344786Z ok (2.658s)
2023-01-11T22:03:10.4344792Z 
2023-01-11T22:03:10.4344983Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4345060Z Ran 1 test in 2.658s
2023-01-11T22:03:10.4345066Z 
2023-01-11T22:03:10.4345130Z OK
2023-01-11T22:03:10.4345138Z 
2023-01-11T22:03:10.4345225Z Generating XML reports...
2023-01-11T22:03:10.4345623Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220001.xml
2023-01-11T22:03:10.4345902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4346031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4346317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4346456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4346629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyinlqvbd
2023-01-11T22:03:10.4346831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyinlqvbd/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4347093Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4347099Z 
2023-01-11T22:03:10.4347173Z Running tests...
2023-01-11T22:03:10.4347373Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4347644Z   test_dynamic_rpc_init_rpc_without_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25425
2023-01-11T22:03:10.4347806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25426
2023-01-11T22:03:10.4347961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25427
2023-01-11T22:03:10.4348102Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25428
2023-01-11T22:03:10.4348423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4348548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4348870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4349008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4349194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmsa40tnk
2023-01-11T22:03:10.4349392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmsa40tnk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4349665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4349791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4350056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4350196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4350385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqffhdrci
2023-01-11T22:03:10.4350582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqffhdrci/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4350858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4350984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4351265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4351401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4351571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8d4chvxy
2023-01-11T22:03:10.4351770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8d4chvxy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4352043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4352168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4352448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4352585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4352773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvnzarso
2023-01-11T22:03:10.4352969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvnzarso/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4353135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4353289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4353455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4353618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4353686Z ok (2.351s)
2023-01-11T22:03:10.4353692Z 
2023-01-11T22:03:10.4353891Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4353970Z Ran 1 test in 2.352s
2023-01-11T22:03:10.4353975Z 
2023-01-11T22:03:10.4354039Z OK
2023-01-11T22:03:10.4354044Z 
2023-01-11T22:03:10.4354129Z Generating XML reports...
2023-01-11T22:03:10.4354509Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220006.xml
2023-01-11T22:03:10.4354785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4354952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4355262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4355400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4355588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpca1mc0i4
2023-01-11T22:03:10.4355787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpca1mc0i4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4356047Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4356053Z 
2023-01-11T22:03:10.4356130Z Running tests...
2023-01-11T22:03:10.4356315Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4356611Z   test_dynamic_rpc_new_rank_can_communicated_with_existing_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25484
2023-01-11T22:03:10.4356772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25485
2023-01-11T22:03:10.4356928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25486
2023-01-11T22:03:10.4357086Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25487
2023-01-11T22:03:10.4357361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4357484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4357771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4357905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4358080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaw47hr9p
2023-01-11T22:03:10.4358280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaw47hr9p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4358555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4358679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4359033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4359173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4359359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpicxfqbwh
2023-01-11T22:03:10.4359557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpicxfqbwh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4359817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4359943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4360225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4360360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4360544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumq7fk0r
2023-01-11T22:03:10.4360817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4360943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4361225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4361359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4361534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthv_hk1v
2023-01-11T22:03:10.4361768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumq7fk0r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4361990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthv_hk1v/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4362157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4362323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4362488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4362654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4362834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4363011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4363175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4363358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4363663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4363968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4364263Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4364556Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4364834Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4365107Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4365375Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4365643Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4365905Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4366153Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4366227Z ok (2.948s)
2023-01-11T22:03:10.4366234Z 
2023-01-11T22:03:10.4366436Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4366514Z Ran 1 test in 2.948s
2023-01-11T22:03:10.4366520Z 
2023-01-11T22:03:10.4366582Z OK
2023-01-11T22:03:10.4366587Z 
2023-01-11T22:03:10.4366673Z Generating XML reports...
2023-01-11T22:03:10.4367070Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220010.xml
2023-01-11T22:03:10.4367352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4367482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4367755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4367924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4368153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgp8hfimm
2023-01-11T22:03:10.4368355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgp8hfimm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4368613Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4368619Z 
2023-01-11T22:03:10.4368695Z Running tests...
2023-01-11T22:03:10.4368896Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4369160Z   test_infer_backend_from_options (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25707
2023-01-11T22:03:10.4369318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25708
2023-01-11T22:03:10.4369465Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25709
2023-01-11T22:03:10.4369622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25710
2023-01-11T22:03:10.4369900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4370025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4370308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4370449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4370637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mg8m1bu
2023-01-11T22:03:10.4370834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mg8m1bu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4371096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4371225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4371512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4371648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4371833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lref91b
2023-01-11T22:03:10.4372029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lref91b/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4372302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4372428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4372713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4372838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4373027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpry_1e4lp
2023-01-11T22:03:10.4373222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpry_1e4lp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4373497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4373621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4373904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4374041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4374226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6e7m_l60
2023-01-11T22:03:10.4374472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6e7m_l60/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4374664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4374832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4374996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4375160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4375434Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4375704Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4375975Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4376048Z ok (2.753s)
2023-01-11T22:03:10.4376054Z 
2023-01-11T22:03:10.4376252Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4376316Z Ran 1 test in 2.754s
2023-01-11T22:03:10.4376321Z 
2023-01-11T22:03:10.4376382Z OK
2023-01-11T22:03:10.4376388Z 
2023-01-11T22:03:10.4376474Z Generating XML reports...
2023-01-11T22:03:10.4376866Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220015.xml
2023-01-11T22:03:10.4377143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4377268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4377556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4377697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4377871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83swupz6
2023-01-11T22:03:10.4378072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83swupz6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4378329Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4378334Z 
2023-01-11T22:03:10.4378409Z Running tests...
2023-01-11T22:03:10.4378607Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4378873Z   test_mismatched_type_for_options (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25930
2023-01-11T22:03:10.4379032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25931
2023-01-11T22:03:10.4379192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25932
2023-01-11T22:03:10.4379349Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25933
2023-01-11T22:03:10.4379610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4379736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4380022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4380158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4380345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9q6br6h
2023-01-11T22:03:10.4380542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9q6br6h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4380851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4381005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4381289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4381412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4381596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw2v64k1d
2023-01-11T22:03:10.4381901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw2v64k1d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4382181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4382307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4382590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4382729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4382919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnohpbrd1
2023-01-11T22:03:10.4383101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnohpbrd1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4383372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4383497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4383778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4383914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4384099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp902hxlwu
2023-01-11T22:03:10.4384296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp902hxlwu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4384466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4384633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4384783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4384948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4385020Z ok (2.347s)
2023-01-11T22:03:10.4385025Z 
2023-01-11T22:03:10.4385225Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4385303Z Ran 1 test in 2.347s
2023-01-11T22:03:10.4385309Z 
2023-01-11T22:03:10.4385370Z OK
2023-01-11T22:03:10.4385375Z 
2023-01-11T22:03:10.4385461Z Generating XML reports...
2023-01-11T22:03:10.4385855Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220020.xml
2023-01-11T22:03:10.4386118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4386245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4386530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4386667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4386857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzn787kz1
2023-01-11T22:03:10.4387054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzn787kz1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4387314Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4387384Z 
2023-01-11T22:03:10.4387456Z Running tests...
2023-01-11T22:03:10.4387655Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4387947Z   test_multi_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25985
2023-01-11T22:03:10.4388110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25986
2023-01-11T22:03:10.4388265Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25987
2023-01-11T22:03:10.4388421Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25988
2023-01-11T22:03:10.4388698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4388824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4389106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4389245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4389433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthscrjza
2023-01-11T22:03:10.4389619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthscrjza/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4389893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4390018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4390302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4390438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4390624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6asm9823
2023-01-11T22:03:10.4390879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6asm9823/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4391157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4391267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4391549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4391684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4391869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjveh1esw
2023-01-11T22:03:10.4392065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjveh1esw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4392337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4392466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4392748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4392882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4393054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pqjxa8f
2023-01-11T22:03:10.4393250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pqjxa8f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4393417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4393584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4393747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4393908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4394007Z ok (2.839s)
2023-01-11T22:03:10.4394013Z 
2023-01-11T22:03:10.4394239Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4394303Z Ran 1 test in 2.839s
2023-01-11T22:03:10.4394309Z 
2023-01-11T22:03:10.4394372Z OK
2023-01-11T22:03:10.4394377Z 
2023-01-11T22:03:10.4394465Z Generating XML reports...
2023-01-11T22:03:10.4394862Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220024.xml
2023-01-11T22:03:10.4395138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4395264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4395547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4395685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4395875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwij0zl0j
2023-01-11T22:03:10.4396061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwij0zl0j/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4396323Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4396328Z 
2023-01-11T22:03:10.4396403Z Running tests...
2023-01-11T22:03:10.4396602Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4396868Z   test_multi_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26220
2023-01-11T22:03:10.4397028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26221
2023-01-11T22:03:10.4397184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26222
2023-01-11T22:03:10.4397342Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26223
2023-01-11T22:03:10.4397609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4397736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4398017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4398155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4398341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmonaci5s
2023-01-11T22:03:10.4398538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmonaci5s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4398814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4399038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4399323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4399447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4399632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcy737os
2023-01-11T22:03:10.4399826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcy737os/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4400099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4400223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4400504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4400639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4400866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahkrmcze
2023-01-11T22:03:10.4401092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahkrmcze/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4401351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4401476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4401753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4401890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4402075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv599ttrt
2023-01-11T22:03:10.4402270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv599ttrt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4402437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4402608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4402758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4402924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4402994Z ok (2.752s)
2023-01-11T22:03:10.4403000Z 
2023-01-11T22:03:10.4403199Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4403276Z Ran 1 test in 2.752s
2023-01-11T22:03:10.4403281Z 
2023-01-11T22:03:10.4403343Z OK
2023-01-11T22:03:10.4403348Z 
2023-01-11T22:03:10.4403436Z Generating XML reports...
2023-01-11T22:03:10.4403833Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220029.xml
2023-01-11T22:03:10.4404114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4404228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4404514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4404655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4404843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkv551oki
2023-01-11T22:03:10.4405040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkv551oki/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4405297Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4405303Z 
2023-01-11T22:03:10.4405378Z Running tests...
2023-01-11T22:03:10.4405578Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4405821Z   test_multi_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26455
2023-01-11T22:03:10.4405984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26456
2023-01-11T22:03:10.4406139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26457
2023-01-11T22:03:10.4406294Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26458
2023-01-11T22:03:10.4406570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4406696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4406984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4407125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4407349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7m4ttd5s
2023-01-11T22:03:10.4407565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7m4ttd5s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4407842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4407967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4408250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4408387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4408574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfyrux7l
2023-01-11T22:03:10.4408773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfyrux7l/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4409048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4409161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4409445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4409585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4409772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5xjman_
2023-01-11T22:03:10.4409968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5xjman_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4410242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4410368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4410649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4410787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4410962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3mv7g_s
2023-01-11T22:03:10.4411160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3mv7g_s/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4411325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4411492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4411657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4411820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4411892Z ok (3.450s)
2023-01-11T22:03:10.4411897Z 
2023-01-11T22:03:10.4412096Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4412161Z Ran 1 test in 3.450s
2023-01-11T22:03:10.4412181Z 
2023-01-11T22:03:10.4412230Z OK
2023-01-11T22:03:10.4412235Z 
2023-01-11T22:03:10.4412323Z Generating XML reports...
2023-01-11T22:03:10.4412713Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220034.xml
2023-01-11T22:03:10.4412989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4413114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4413397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4413533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4413720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk45xn7n0
2023-01-11T22:03:10.4413932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk45xn7n0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4414232Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4414238Z 
2023-01-11T22:03:10.4414314Z Running tests...
2023-01-11T22:03:10.4414514Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4414781Z   test_my_parameter_server_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26690
2023-01-11T22:03:10.4414939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26691
2023-01-11T22:03:10.4415096Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26692
2023-01-11T22:03:10.4415248Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26693
2023-01-11T22:03:10.4415527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4415642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4415926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4416062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4416249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe945oa92
2023-01-11T22:03:10.4416444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe945oa92/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4416724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4416850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4417132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4417257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4417445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpugbi9n95
2023-01-11T22:03:10.4417640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpugbi9n95/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4417913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4418036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4418319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4418456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4418642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplnlylddf
2023-01-11T22:03:10.4418839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplnlylddf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4419105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4419231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4419514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4419650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4419836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthh7oet4
2023-01-11T22:03:10.4420031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthh7oet4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4420198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4420365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4420551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4420750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4420820Z ok (2.856s)
2023-01-11T22:03:10.4420826Z 
2023-01-11T22:03:10.4421029Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4421105Z Ran 1 test in 2.856s
2023-01-11T22:03:10.4421111Z 
2023-01-11T22:03:10.4421173Z OK
2023-01-11T22:03:10.4421178Z 
2023-01-11T22:03:10.4421263Z Generating XML reports...
2023-01-11T22:03:10.4421755Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220039.xml
2023-01-11T22:03:10.4422030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4422142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4422428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4422568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4422755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdytyofl0
2023-01-11T22:03:10.4422956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdytyofl0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4423214Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4423220Z 
2023-01-11T22:03:10.4423294Z Running tests...
2023-01-11T22:03:10.4423494Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4423754Z   test_nested_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26949
2023-01-11T22:03:10.4423902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26950
2023-01-11T22:03:10.4424059Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26951
2023-01-11T22:03:10.4424215Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26952
2023-01-11T22:03:10.4424492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4424616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4424900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4425038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4425224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpke5wjom9
2023-01-11T22:03:10.4425407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpke5wjom9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4425684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4425811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4426096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4426235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4426419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0_561co
2023-01-11T22:03:10.4426671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0_561co/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4426945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4427070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4427392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4427560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4427749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ujm4ys4
2023-01-11T22:03:10.4427943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ujm4ys4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4428219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4428345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4428627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4428765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4428936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyqzviz71
2023-01-11T22:03:10.4429135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqzviz71/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4429301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4429467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4429632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4429793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4429862Z ok (2.641s)
2023-01-11T22:03:10.4429868Z 
2023-01-11T22:03:10.4430066Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4430143Z Ran 1 test in 2.642s
2023-01-11T22:03:10.4430149Z 
2023-01-11T22:03:10.4430198Z OK
2023-01-11T22:03:10.4430202Z 
2023-01-11T22:03:10.4430289Z Generating XML reports...
2023-01-11T22:03:10.4430683Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220044.xml
2023-01-11T22:03:10.4430965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4431090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4431374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4431511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4431699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhpijvtz
2023-01-11T22:03:10.4431885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhpijvtz/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4432145Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4432153Z 
2023-01-11T22:03:10.4432228Z Running tests...
2023-01-11T22:03:10.4432426Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4432684Z   test_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27196
2023-01-11T22:03:10.4432843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27197
2023-01-11T22:03:10.4433000Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27198
2023-01-11T22:03:10.4433155Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27199
2023-01-11T22:03:10.4433431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4433543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4433825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4433994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4434209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2i51xbqg
2023-01-11T22:03:10.4434409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2i51xbqg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4434684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4434812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4435095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4435219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4435405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpezz5paf3
2023-01-11T22:03:10.4435605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpezz5paf3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4435883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4436009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4436293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4436428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4436616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5bgi3g8
2023-01-11T22:03:10.4436811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5bgi3g8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4437070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4437201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4437484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4437621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4437805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqk_chi_n
2023-01-11T22:03:10.4438001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqk_chi_n/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4438167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4438334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4438497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4438647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4438719Z ok (2.740s)
2023-01-11T22:03:10.4438724Z 
2023-01-11T22:03:10.4438997Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4439076Z Ran 1 test in 2.741s
2023-01-11T22:03:10.4439081Z 
2023-01-11T22:03:10.4439144Z OK
2023-01-11T22:03:10.4439148Z 
2023-01-11T22:03:10.4439236Z Generating XML reports...
2023-01-11T22:03:10.4439634Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220048.xml
2023-01-11T22:03:10.4439911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4440022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4440308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4440483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4440672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_5i2to5
2023-01-11T22:03:10.4440894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_5i2to5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4441154Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4441159Z 
2023-01-11T22:03:10.4441234Z Running tests...
2023-01-11T22:03:10.4441435Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4441691Z   test_nested_rref_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27443
2023-01-11T22:03:10.4441835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27444
2023-01-11T22:03:10.4441994Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27445
2023-01-11T22:03:10.4442151Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27446
2023-01-11T22:03:10.4442431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4442559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4442842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4442981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4443166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12r_je9x
2023-01-11T22:03:10.4443346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12r_je9x/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4443622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4443751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4444037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4444173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4444358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptv08lqlm
2023-01-11T22:03:10.4444554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptv08lqlm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4444829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4444955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4445225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4445361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4445548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtz9asmq
2023-01-11T22:03:10.4445747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtz9asmq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4446020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4446146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4446427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4446564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4446736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0y3oo_v2
2023-01-11T22:03:10.4446932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0y3oo_v2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4447134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4447328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4447492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4447656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4447726Z ok (2.740s)
2023-01-11T22:03:10.4447731Z 
2023-01-11T22:03:10.4447930Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4448007Z Ran 1 test in 2.740s
2023-01-11T22:03:10.4448013Z 
2023-01-11T22:03:10.4448061Z OK
2023-01-11T22:03:10.4448066Z 
2023-01-11T22:03:10.4448153Z Generating XML reports...
2023-01-11T22:03:10.4448543Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220053.xml
2023-01-11T22:03:10.4448824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4448952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4449233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4449370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4449556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvk3xbuhg
2023-01-11T22:03:10.4449738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvk3xbuhg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4449995Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4450000Z 
2023-01-11T22:03:10.4450075Z Running tests...
2023-01-11T22:03:10.4450272Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4450540Z   test_nested_rref_stress_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27690
2023-01-11T22:03:10.4450702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27691
2023-01-11T22:03:10.4450860Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27692
2023-01-11T22:03:10.4451015Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27693
2023-01-11T22:03:10.4451291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4451401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4451688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4451823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4452014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3cm1yd36
2023-01-11T22:03:10.4452212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3cm1yd36/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4452489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4452616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4452902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4453040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4453213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmall87t4
2023-01-11T22:03:10.4453410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmall87t4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4453714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4453840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4454150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4454287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4454474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkys3m1c
2023-01-11T22:03:10.4454671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkys3m1c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4454933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4455060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4455341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4455481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4455668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpct9nk_t1
2023-01-11T22:03:10.4455863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpct9nk_t1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4456030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4456197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4456361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4456510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4456579Z ok (2.845s)
2023-01-11T22:03:10.4456585Z 
2023-01-11T22:03:10.4456783Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4456863Z Ran 1 test in 2.845s
2023-01-11T22:03:10.4456868Z 
2023-01-11T22:03:10.4456931Z OK
2023-01-11T22:03:10.4456936Z 
2023-01-11T22:03:10.4457023Z Generating XML reports...
2023-01-11T22:03:10.4457420Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220058.xml
2023-01-11T22:03:10.4457699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4457810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4458093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4458229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4458416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvcgtzdd8
2023-01-11T22:03:10.4458616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvcgtzdd8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4458875Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4458881Z 
2023-01-11T22:03:10.4458956Z Running tests...
2023-01-11T22:03:10.4459157Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4459416Z   test_op_with_invalid_args (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28117
2023-01-11T22:03:10.4459561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28118
2023-01-11T22:03:10.4459717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28119
2023-01-11T22:03:10.4459871Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28120
2023-01-11T22:03:10.4460148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4460310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4460632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4460772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4460961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdvspsclg
2023-01-11T22:03:10.4461159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdvspsclg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4461422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4461547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4461929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4462069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4462255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hoh6s5g
2023-01-11T22:03:10.4462451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hoh6s5g/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4462725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4462849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4463114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4463251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4463437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9ohoi0c
2023-01-11T22:03:10.4463635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9ohoi0c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4463910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4464035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4464313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4464449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4464633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4m73sr8k
2023-01-11T22:03:10.4464813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4m73sr8k/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4464979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4465146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4465315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4465477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4465547Z ok (2.741s)
2023-01-11T22:03:10.4465553Z 
2023-01-11T22:03:10.4465751Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4465827Z Ran 1 test in 2.741s
2023-01-11T22:03:10.4465833Z 
2023-01-11T22:03:10.4465882Z OK
2023-01-11T22:03:10.4465887Z 
2023-01-11T22:03:10.4465975Z Generating XML reports...
2023-01-11T22:03:10.4466370Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220103.xml
2023-01-11T22:03:10.4466647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4466772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4467107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4467277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4467466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpna5f2hry
2023-01-11T22:03:10.4467663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpna5f2hry/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4467909Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4467914Z 
2023-01-11T22:03:10.4467990Z Running tests...
2023-01-11T22:03:10.4468187Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4468449Z   test_py_rpc_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28340
2023-01-11T22:03:10.4468609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28341
2023-01-11T22:03:10.4468767Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28342
2023-01-11T22:03:10.4468924Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28343
2023-01-11T22:03:10.4469202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4469315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4469599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4469737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4469923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_0vimwh
2023-01-11T22:03:10.4470119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_0vimwh/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4470395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4470524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4470810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4470949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4471123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmb9_08t
2023-01-11T22:03:10.4471319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmb9_08t/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4471590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4471720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4472003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4472141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4472328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg97le_03
2023-01-11T22:03:10.4472520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg97le_03/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4472794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4472904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4473185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4473321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4473538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48f9piq0
2023-01-11T22:03:10.4473793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48f9piq0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4473959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4474125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4474288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4474438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4474508Z ok (2.742s)
2023-01-11T22:03:10.4474514Z 
2023-01-11T22:03:10.4474716Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4474793Z Ran 1 test in 2.743s
2023-01-11T22:03:10.4474799Z 
2023-01-11T22:03:10.4474862Z OK
2023-01-11T22:03:10.4474870Z 
2023-01-11T22:03:10.4474955Z Generating XML reports...
2023-01-11T22:03:10.4475353Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220107.xml
2023-01-11T22:03:10.4475629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4475755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4476027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4476165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4476350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpanb9nrv6
2023-01-11T22:03:10.4476548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpanb9nrv6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4476807Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4476815Z 
2023-01-11T22:03:10.4476890Z Running tests...
2023-01-11T22:03:10.4477090Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4477346Z   test_py_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28575
2023-01-11T22:03:10.4477493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28576
2023-01-11T22:03:10.4477649Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28577
2023-01-11T22:03:10.4477804Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28578
2023-01-11T22:03:10.4478082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4478210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4478499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4478639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4478895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgu9qkvo5
2023-01-11T22:03:10.4479096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgu9qkvo5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4479361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4479490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4479775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4479909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4480095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2j69gl2a
2023-01-11T22:03:10.4480325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2j69gl2a/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4480626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4480753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4481021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4481157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4481344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu446pl8p
2023-01-11T22:03:10.4481543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu446pl8p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4481816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4481946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4482228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4482369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4482554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvqtchzar
2023-01-11T22:03:10.4482735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvqtchzar/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4482902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4483067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4483230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4483396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4483467Z ok (2.747s)
2023-01-11T22:03:10.4483472Z 
2023-01-11T22:03:10.4483673Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4483751Z Ran 1 test in 2.747s
2023-01-11T22:03:10.4483757Z 
2023-01-11T22:03:10.4483804Z OK
2023-01-11T22:03:10.4483823Z 
2023-01-11T22:03:10.4483895Z Generating XML reports...
2023-01-11T22:03:10.4484290Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220112.xml
2023-01-11T22:03:10.4484568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4484695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4484975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4485114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4485306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnbvnpzr
2023-01-11T22:03:10.4485503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnbvnpzr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4485745Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4485765Z 
2023-01-11T22:03:10.4485827Z Running tests...
2023-01-11T22:03:10.4486025Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4486292Z   test_py_rref_args_user_share_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28810
2023-01-11T22:03:10.4486452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28811
2023-01-11T22:03:10.4486609Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28812
2023-01-11T22:03:10.4486795Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28813
2023-01-11T22:03:10.4487098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4487224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4487495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4487631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4487818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdr5xb154
2023-01-11T22:03:10.4488016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdr5xb154/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4488291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4488421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4488704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4488840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4489012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb81_6bcs
2023-01-11T22:03:10.4489208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb81_6bcs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4489481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4489607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4489893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4490034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4490219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatgndyrf
2023-01-11T22:03:10.4490420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatgndyrf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4490693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4490803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4491088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4491224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4491409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyir1iadc
2023-01-11T22:03:10.4491605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyir1iadc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4491774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4491939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4492104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4492254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4492325Z ok (2.843s)
2023-01-11T22:03:10.4492331Z 
2023-01-11T22:03:10.4492528Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4492607Z Ran 1 test in 2.843s
2023-01-11T22:03:10.4492613Z 
2023-01-11T22:03:10.4492676Z OK
2023-01-11T22:03:10.4492681Z 
2023-01-11T22:03:10.4492767Z Generating XML reports...
2023-01-11T22:03:10.4493159Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220117.xml
2023-01-11T22:03:10.4493468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4493621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4493894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4494036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4494223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuks3z17w
2023-01-11T22:03:10.4494420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuks3z17w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4494678Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4494684Z 
2023-01-11T22:03:10.4494759Z Running tests...
2023-01-11T22:03:10.4494957Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4495232Z   test_py_sparse_tensors_in_container (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29045
2023-01-11T22:03:10.4495391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29046
2023-01-11T22:03:10.4495539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29047
2023-01-11T22:03:10.4495695Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29048
2023-01-11T22:03:10.4495973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4496099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4496385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4496522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4496711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgtlutzxm
2023-01-11T22:03:10.4496958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgtlutzxm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4497309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4497439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4497723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4497859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4498044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpai9ib2kv
2023-01-11T22:03:10.4498240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpai9ib2kv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4498517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4498644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4498929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4499053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4499239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9897pza
2023-01-11T22:03:10.4499516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4499640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4499923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4500098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4500285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd6zi4x1r
2023-01-11T22:03:10.4500513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9897pza/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4500694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd6zi4x1r/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4500861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4501027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4501191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4501353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4501423Z ok (2.741s)
2023-01-11T22:03:10.4501429Z 
2023-01-11T22:03:10.4501636Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4501803Z Ran 1 test in 2.741s
2023-01-11T22:03:10.4501810Z 
2023-01-11T22:03:10.4501873Z OK
2023-01-11T22:03:10.4501880Z 
2023-01-11T22:03:10.4501952Z Generating XML reports...
2023-01-11T22:03:10.4502349Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220121.xml
2023-01-11T22:03:10.4502624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4502748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4503031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4503167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4503357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodotjvep
2023-01-11T22:03:10.4503557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodotjvep/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4503802Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4503822Z 
2023-01-11T22:03:10.4503884Z Running tests...
2023-01-11T22:03:10.4504083Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4504352Z   test_rref_get_type_timeout_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29280
2023-01-11T22:03:10.4504513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29281
2023-01-11T22:03:10.4504668Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29282
2023-01-11T22:03:10.4504825Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29283
2023-01-11T22:03:10.4505106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4505233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4505508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4505647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4505835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptz_rk_c9
2023-01-11T22:03:10.4506031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptz_rk_c9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4506307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4506433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4506717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4506910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4507148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0894kqpy
2023-01-11T22:03:10.4507331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0894kqpy/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4507606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4507732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4508014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4508149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4508334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpll4xliq1
2023-01-11T22:03:10.4508534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpll4xliq1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4508811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4508922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4509204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4509340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4509526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmv0vzm1b
2023-01-11T22:03:10.4509725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmv0vzm1b/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4509892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4510061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4510224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4510389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4510446Z ok (4.647s)
2023-01-11T22:03:10.4510452Z 
2023-01-11T22:03:10.4510649Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4510727Z Ran 1 test in 4.647s
2023-01-11T22:03:10.4510733Z 
2023-01-11T22:03:10.4510794Z OK
2023-01-11T22:03:10.4510799Z 
2023-01-11T22:03:10.4510884Z Generating XML reports...
2023-01-11T22:03:10.4511276Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220126.xml
2023-01-11T22:03:10.4511552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4511681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4511951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4512088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4512275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc57n_21u
2023-01-11T22:03:10.4512470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc57n_21u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4512728Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4512733Z 
2023-01-11T22:03:10.4512807Z Running tests...
2023-01-11T22:03:10.4513007Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4513280Z   test_rref_get_type_timeout_non_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29503
2023-01-11T22:03:10.4513470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29504
2023-01-11T22:03:10.4513639Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29505
2023-01-11T22:03:10.4513798Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29506
2023-01-11T22:03:10.4514075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4514200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4514486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4514624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4514811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg80ukrhl
2023-01-11T22:03:10.4515012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg80ukrhl/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4515287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4515399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4515681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4515817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4516003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfw67pamj
2023-01-11T22:03:10.4516201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfw67pamj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4516472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4516599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4516884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4517004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4517189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2k6cwpgx
2023-01-11T22:03:10.4517461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4517586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4517868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4518003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4518188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hzhbxj4
2023-01-11T22:03:10.4518387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2k6cwpgx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4518581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hzhbxj4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4518735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4518973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4519137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4519301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4519370Z ok (4.644s)
2023-01-11T22:03:10.4519375Z 
2023-01-11T22:03:10.4519583Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4519660Z Ran 1 test in 4.644s
2023-01-11T22:03:10.4519700Z 
2023-01-11T22:03:10.4519763Z OK
2023-01-11T22:03:10.4519768Z 
2023-01-11T22:03:10.4519839Z Generating XML reports...
2023-01-11T22:03:10.4520262Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220133.xml
2023-01-11T22:03:10.4520541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4520669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4520956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4521093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4521282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbe8fe8i9
2023-01-11T22:03:10.4521481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbe8fe8i9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4521739Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4521745Z 
2023-01-11T22:03:10.4521805Z Running tests...
2023-01-11T22:03:10.4522006Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4522263Z   test_rref_proxy_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29726
2023-01-11T22:03:10.4522426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29727
2023-01-11T22:03:10.4522584Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29728
2023-01-11T22:03:10.4522741Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29729
2023-01-11T22:03:10.4523017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4523142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4523412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4523552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4523739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmehipfid
2023-01-11T22:03:10.4523938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmehipfid/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4524211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4524335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4524618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4524754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4524941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3owuq1zn
2023-01-11T22:03:10.4525123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3owuq1zn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4525398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4525522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4525800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4525935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4526121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphemd8l3m
2023-01-11T22:03:10.4526316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphemd8l3m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4526592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4526734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4527047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4527186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4527371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe11cawbq
2023-01-11T22:03:10.4527564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe11cawbq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4527729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4527894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4528057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4528222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4528485Z [W tensorpipe_agent.cpp:939] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4528775Z [W tensorpipe_agent.cpp:624] RPC agent for worker1 won't send response to request #30 to worker0, as the agent is shutting down
2023-01-11T22:03:10.4529060Z [W tensorpipe_agent.cpp:624] RPC agent for worker0 won't send response to request #28 to worker3, as the agent is shutting down
2023-01-11T22:03:10.4529340Z [W tensorpipe_agent.cpp:624] RPC agent for worker3 won't send response to request #28 to worker2, as the agent is shutting down
2023-01-11T22:03:10.4529621Z [W tensorpipe_agent.cpp:624] RPC agent for worker2 won't send response to request #30 to worker1, as the agent is shutting down
2023-01-11T22:03:10.4529695Z ok (15.667s)
2023-01-11T22:03:10.4529700Z 
2023-01-11T22:03:10.4529900Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4529977Z Ran 1 test in 15.667s
2023-01-11T22:03:10.4529985Z 
2023-01-11T22:03:10.4530050Z OK
2023-01-11T22:03:10.4530056Z 
2023-01-11T22:03:10.4530127Z Generating XML reports...
2023-01-11T22:03:10.4530519Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220139.xml
2023-01-11T22:03:10.4530795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4530919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4531205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4531341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4531531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_xzpjzv
2023-01-11T22:03:10.4531728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_xzpjzv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4531985Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4531990Z 
2023-01-11T22:03:10.4532050Z Running tests...
2023-01-11T22:03:10.4532247Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4532511Z   test_self_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29949
2023-01-11T22:03:10.4532669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29950
2023-01-11T22:03:10.4532826Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29951
2023-01-11T22:03:10.4532981Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29952
2023-01-11T22:03:10.4533291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4533445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4533732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4533857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4534043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5z49nfs5
2023-01-11T22:03:10.4534239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5z49nfs5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4534517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4534641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4534927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4535066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4535252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpad29gi0f
2023-01-11T22:03:10.4535435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpad29gi0f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4535707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4535832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4536113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4536250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4536438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppxeveprf
2023-01-11T22:03:10.4536713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4536838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4537120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4537244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4537432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswy2nkn4
2023-01-11T22:03:10.4537630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppxeveprf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4537820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswy2nkn4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4537985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4538153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4538319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4538481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4538536Z ok (2.744s)
2023-01-11T22:03:10.4538555Z 
2023-01-11T22:03:10.4538739Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4538814Z Ran 1 test in 2.744s
2023-01-11T22:03:10.4538820Z 
2023-01-11T22:03:10.4538882Z OK
2023-01-11T22:03:10.4538887Z 
2023-01-11T22:03:10.4538973Z Generating XML reports...
2023-01-11T22:03:10.4539365Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220157.xml
2023-01-11T22:03:10.4539643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4539803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4540112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4540236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4540422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fod8i2g
2023-01-11T22:03:10.4540620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fod8i2g/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4540877Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4540882Z 
2023-01-11T22:03:10.4540956Z Running tests...
2023-01-11T22:03:10.4541153Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4541427Z   test_self_remote_rref_as_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30184
2023-01-11T22:03:10.4541589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30185
2023-01-11T22:03:10.4541841Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30186
2023-01-11T22:03:10.4541982Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30187
2023-01-11T22:03:10.4542263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4542388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4542672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4542809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4542996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmping8yt9c
2023-01-11T22:03:10.4543197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmping8yt9c/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4543473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4543586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4543869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4544005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4544193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpltucpx0q
2023-01-11T22:03:10.4544391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpltucpx0q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4544661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4544787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4545069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4545207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4545377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1d9_navt
2023-01-11T22:03:10.4545572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1d9_navt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4545843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4545969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4546249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4546435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4546652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_i9h2cg
2023-01-11T22:03:10.4546848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_i9h2cg/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4547000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4547168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4547333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4547496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4547566Z ok (2.759s)
2023-01-11T22:03:10.4547571Z 
2023-01-11T22:03:10.4547773Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4547853Z Ran 1 test in 2.759s
2023-01-11T22:03:10.4547859Z 
2023-01-11T22:03:10.4547922Z OK
2023-01-11T22:03:10.4547928Z 
2023-01-11T22:03:10.4548013Z Generating XML reports...
2023-01-11T22:03:10.4548395Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220201.xml
2023-01-11T22:03:10.4548672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4548797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4549077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4549214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4549399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsk2sv3gs
2023-01-11T22:03:10.4549595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsk2sv3gs/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4549856Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4549862Z 
2023-01-11T22:03:10.4549938Z Running tests...
2023-01-11T22:03:10.4550124Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4550396Z   test_self_remote_rref_as_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30419
2023-01-11T22:03:10.4550555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30420
2023-01-11T22:03:10.4550711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30421
2023-01-11T22:03:10.4550866Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30422
2023-01-11T22:03:10.4551143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4551270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4551559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4551682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4551872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg31g29wp
2023-01-11T22:03:10.4552068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg31g29wp/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4552343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4552471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4552757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4552923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4553115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx92gjjvn
2023-01-11T22:03:10.4553350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx92gjjvn/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4553612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4553738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4554019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4554157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4554344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmehjr9be
2023-01-11T22:03:10.4554541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmehjr9be/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4554816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4554943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4555211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4555348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4555533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqnee84p8
2023-01-11T22:03:10.4555728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqnee84p8/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4555893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4556061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4556227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4556393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4556461Z ok (2.749s)
2023-01-11T22:03:10.4556467Z 
2023-01-11T22:03:10.4556652Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4556730Z Ran 1 test in 2.749s
2023-01-11T22:03:10.4556736Z 
2023-01-11T22:03:10.4556803Z OK
2023-01-11T22:03:10.4556808Z 
2023-01-11T22:03:10.4556893Z Generating XML reports...
2023-01-11T22:03:10.4557280Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220206.xml
2023-01-11T22:03:10.4557558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4557685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4557971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4558098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4558290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcxgbj_9p
2023-01-11T22:03:10.4558488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcxgbj_9p/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4558745Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4558750Z 
2023-01-11T22:03:10.4558898Z Running tests...
2023-01-11T22:03:10.4559101Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4559381Z   test_self_remote_rref_as_self_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30654
2023-01-11T22:03:10.4559577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30655
2023-01-11T22:03:10.4559736Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30656
2023-01-11T22:03:10.4559905Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30657
2023-01-11T22:03:10.4560184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4560311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4560593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4560730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4560919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kvhumib
2023-01-11T22:03:10.4561118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kvhumib/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4561396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4561525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4561793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4561929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4562115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6cpzslc
2023-01-11T22:03:10.4562313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6cpzslc/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4562586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4562712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4562995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4563134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4563304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4n0vpb1z
2023-01-11T22:03:10.4563502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4n0vpb1z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4563772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4563898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4564177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4564313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4564501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fedq83i
2023-01-11T22:03:10.4564700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fedq83i/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4564869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4565020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4565182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4565345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4565415Z ok (2.646s)
2023-01-11T22:03:10.4565421Z 
2023-01-11T22:03:10.4565620Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4565699Z Ran 1 test in 2.647s
2023-01-11T22:03:10.4565705Z 
2023-01-11T22:03:10.4565767Z OK
2023-01-11T22:03:10.4565772Z 
2023-01-11T22:03:10.4565857Z Generating XML reports...
2023-01-11T22:03:10.4566285Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220211.xml
2023-01-11T22:03:10.4566586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4566715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4566996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4567135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4567322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnt36m0z
2023-01-11T22:03:10.4567520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnt36m0z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4567777Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4567785Z 
2023-01-11T22:03:10.4567860Z Running tests...
2023-01-11T22:03:10.4568045Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4568323Z   test_self_remote_rref_as_self_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30889
2023-01-11T22:03:10.4568482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30890
2023-01-11T22:03:10.4568637Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30891
2023-01-11T22:03:10.4568792Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30892
2023-01-11T22:03:10.4569071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4569197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4569481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4569620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4569796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8cehx4mq
2023-01-11T22:03:10.4569993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8cehx4mq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4570270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4570398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4570680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4570818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4571004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiknwngs_
2023-01-11T22:03:10.4571203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiknwngs_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4571465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4571590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4571871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4572007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4572191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8je6z7i4
2023-01-11T22:03:10.4572390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8je6z7i4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4572664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4572823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4573133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4573256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4573441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxio_m5_
2023-01-11T22:03:10.4573634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxio_m5_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4573800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4573967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4574133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4574297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4574370Z ok (2.748s)
2023-01-11T22:03:10.4574375Z 
2023-01-11T22:03:10.4574564Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4574644Z Ran 1 test in 2.748s
2023-01-11T22:03:10.4574649Z 
2023-01-11T22:03:10.4574712Z OK
2023-01-11T22:03:10.4574717Z 
2023-01-11T22:03:10.4574803Z Generating XML reports...
2023-01-11T22:03:10.4575196Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220215.xml
2023-01-11T22:03:10.4575472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4575599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4575882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4576022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4576198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrwaqoq6
2023-01-11T22:03:10.4576399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrwaqoq6/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4576657Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4576663Z 
2023-01-11T22:03:10.4576739Z Running tests...
2023-01-11T22:03:10.4576938Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4577197Z   test_send_to_rank_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31124
2023-01-11T22:03:10.4577359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31125
2023-01-11T22:03:10.4577515Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31126
2023-01-11T22:03:10.4577660Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31127
2023-01-11T22:03:10.4577940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4578067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4578351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4578490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4578677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3ckt7w1
2023-01-11T22:03:10.4578875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3ckt7w1/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4579151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4579339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4579636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4579774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4579959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_svpqck
2023-01-11T22:03:10.4580156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_svpqck/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4580431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4580556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4580839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4580974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4581148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgzsx0fm
2023-01-11T22:03:10.4581347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgzsx0fm/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4581625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4581860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4582146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4582282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4582467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6o0y_4q
2023-01-11T22:03:10.4582659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6o0y_4q/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4582828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4582982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4583144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4583305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4583374Z ok (2.842s)
2023-01-11T22:03:10.4583381Z 
2023-01-11T22:03:10.4583580Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4583657Z Ran 1 test in 2.842s
2023-01-11T22:03:10.4583662Z 
2023-01-11T22:03:10.4583723Z OK
2023-01-11T22:03:10.4583728Z 
2023-01-11T22:03:10.4583814Z Generating XML reports...
2023-01-11T22:03:10.4584196Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220220.xml
2023-01-11T22:03:10.4584476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4584605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4584888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4585025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4585211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9v5b19nx
2023-01-11T22:03:10.4585411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9v5b19nx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4585668Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4585674Z 
2023-01-11T22:03:10.4585749Z Running tests...
2023-01-11T22:03:10.4585936Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4586251Z   test_set_and_get_num_worker_threads (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31359
2023-01-11T22:03:10.4586440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31360
2023-01-11T22:03:10.4586601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31361
2023-01-11T22:03:10.4586756Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31362
2023-01-11T22:03:10.4587038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4587163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4587452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4587589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4587766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3pw008o
2023-01-11T22:03:10.4587964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3pw008o/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4588240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4588366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4588650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4588787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4588972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9kr1cqj
2023-01-11T22:03:10.4589169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9kr1cqj/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4589431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4589556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4589840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4589978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4590163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58vce2xq
2023-01-11T22:03:10.4590356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58vce2xq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4590630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4590755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4591036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4591160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4591348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0us0f50
2023-01-11T22:03:10.4591541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0us0f50/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4591708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4591873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4592036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4592199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4592268Z ok (2.647s)
2023-01-11T22:03:10.4592274Z 
2023-01-11T22:03:10.4592462Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4592577Z Ran 1 test in 2.647s
2023-01-11T22:03:10.4592582Z 
2023-01-11T22:03:10.4592646Z OK
2023-01-11T22:03:10.4592651Z 
2023-01-11T22:03:10.4592763Z Generating XML reports...
2023-01-11T22:03:10.4593164Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220225.xml
2023-01-11T22:03:10.4593440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4593565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4593850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4593986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4594159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ke2p6gf
2023-01-11T22:03:10.4594358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ke2p6gf/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4594618Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4594623Z 
2023-01-11T22:03:10.4594698Z Running tests...
2023-01-11T22:03:10.4594898Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4595159Z   test_stress_heavy_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31626
2023-01-11T22:03:10.4595318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31627
2023-01-11T22:03:10.4595474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31628
2023-01-11T22:03:10.4595630Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31629
2023-01-11T22:03:10.4595894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4596022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4596309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4596447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4596635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv4e8l6j4
2023-01-11T22:03:10.4596831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv4e8l6j4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4597106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4597232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4597502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4597642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4597830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfckfpho0
2023-01-11T22:03:10.4598105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4598228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4598509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4598645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4598906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvu805rfv
2023-01-11T22:03:10.4599105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfckfpho0/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4599322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvu805rfv/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4599641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4599767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4600050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4600184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4600369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pmuo12w
2023-01-11T22:03:10.4600563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pmuo12w/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4600731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4600883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4601049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4601212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4601330Z Rank 0 finished testing 20 times in 0.11337876319885254 seconds.
2023-01-11T22:03:10.4601444Z Rank 3 finished testing 20 times in 0.10093307495117188 seconds.
2023-01-11T22:03:10.4601554Z Rank 2 finished testing 20 times in 0.1321864128112793 seconds.
2023-01-11T22:03:10.4601672Z Rank 1 finished testing 20 times in 0.12411022186279297 seconds.
2023-01-11T22:03:10.4601742Z ok (2.743s)
2023-01-11T22:03:10.4601748Z 
2023-01-11T22:03:10.4601950Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4602027Z Ran 1 test in 2.743s
2023-01-11T22:03:10.4602033Z 
2023-01-11T22:03:10.4602094Z OK
2023-01-11T22:03:10.4602099Z 
2023-01-11T22:03:10.4602188Z Generating XML reports...
2023-01-11T22:03:10.4602581Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220230.xml
2023-01-11T22:03:10.4602844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4602973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4603255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4603392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4603577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_wy_reu
2023-01-11T22:03:10.4603774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_wy_reu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4604030Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4604038Z 
2023-01-11T22:03:10.4604113Z Running tests...
2023-01-11T22:03:10.4604312Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4604584Z   test_tensorpipe_options_throw_on_timedelta_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31861
2023-01-11T22:03:10.4604746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31862
2023-01-11T22:03:10.4604901Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31863
2023-01-11T22:03:10.4605056Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31864
2023-01-11T22:03:10.4605332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4605458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4605780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4605918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4606120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72ah0cov
2023-01-11T22:03:10.4606319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72ah0cov/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4606595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4606721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4607002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4607139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4607327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwff7404f
2023-01-11T22:03:10.4607525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwff7404f/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4607801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4607912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4608194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4608331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4608515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbvq6wgyx
2023-01-11T22:03:10.4608711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbvq6wgyx/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4608987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4609115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4609398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4609533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4609706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5f0dua7
2023-01-11T22:03:10.4609902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5f0dua7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4610068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4610234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4610399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4610564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4610634Z ok (2.444s)
2023-01-11T22:03:10.4610640Z 
2023-01-11T22:03:10.4610842Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4610905Z Ran 1 test in 2.444s
2023-01-11T22:03:10.4610910Z 
2023-01-11T22:03:10.4610974Z OK
2023-01-11T22:03:10.4610979Z 
2023-01-11T22:03:10.4611063Z Generating XML reports...
2023-01-11T22:03:10.4611458Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220234.xml
2023-01-11T22:03:10.4611736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4611860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4612146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4612317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4612502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1jxa_2if
2023-01-11T22:03:10.4612712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1jxa_2if/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4612973Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4612979Z 
2023-01-11T22:03:10.4613052Z Running tests...
2023-01-11T22:03:10.4613249Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4613518Z   test_tensorpipe_set_default_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31920
2023-01-11T22:03:10.4613679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31921
2023-01-11T22:03:10.4613833Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31922
2023-01-11T22:03:10.4613991Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31923
2023-01-11T22:03:10.4614255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4614378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4614661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4614796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4614982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7a7gg4u
2023-01-11T22:03:10.4615176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7a7gg4u/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4615448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4615574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4615855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4615977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4616160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptd700y6z
2023-01-11T22:03:10.4616352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptd700y6z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4616625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4616747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4617028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4617165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4617350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzpt0ctk
2023-01-11T22:03:10.4617545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzpt0ctk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4617804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4617926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4618205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4618338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4618518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp40_sz6rk
2023-01-11T22:03:10.4618708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp40_sz6rk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4618904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4619101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4619252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4619414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4619481Z ok (2.647s)
2023-01-11T22:03:10.4619487Z 
2023-01-11T22:03:10.4619682Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4619757Z Ran 1 test in 2.647s
2023-01-11T22:03:10.4619763Z 
2023-01-11T22:03:10.4619823Z OK
2023-01-11T22:03:10.4619828Z 
2023-01-11T22:03:10.4619911Z Generating XML reports...
2023-01-11T22:03:10.4620301Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220239.xml
2023-01-11T22:03:10.4620580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4620695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4620980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4621116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4621302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplr96hd_2
2023-01-11T22:03:10.4621498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplr96hd_2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4621851Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4621857Z 
2023-01-11T22:03:10.4621933Z Running tests...
2023-01-11T22:03:10.4622134Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4622386Z   test_wait_all_workers_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32143
2023-01-11T22:03:10.4622543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32144
2023-01-11T22:03:10.4622697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32145
2023-01-11T22:03:10.4622852Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32146
2023-01-11T22:03:10.4623128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4623254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4623539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4623674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4623863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5pww2cu
2023-01-11T22:03:10.4624047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5pww2cu/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4624321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4624445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4624726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4624862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4625046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprz9lcbnw
2023-01-11T22:03:10.4625242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprz9lcbnw/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4625560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4625707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4625989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4626125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4626314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdvzlno_
2023-01-11T22:03:10.4626511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdvzlno_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4626786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4626910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4627191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4627329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4627500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vl6o836
2023-01-11T22:03:10.4627691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vl6o836/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4627855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4628021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4628183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4628344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4628520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4628700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4628874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4629037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4629338Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4629634Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4629927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4630215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4630486Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.4630750Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.4631012Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.4631277Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4631540Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4631857Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.4631915Z ok (2.948s)
2023-01-11T22:03:10.4631921Z 
2023-01-11T22:03:10.4632117Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4632192Z Ran 1 test in 2.948s
2023-01-11T22:03:10.4632197Z 
2023-01-11T22:03:10.4632258Z OK
2023-01-11T22:03:10.4632263Z 
2023-01-11T22:03:10.4632346Z Generating XML reports...
2023-01-11T22:03:10.4632738Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220243.xml
2023-01-11T22:03:10.4633014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4633142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4633424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4633551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4633738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt99wput_
2023-01-11T22:03:10.4633937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt99wput_/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4634191Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4634197Z 
2023-01-11T22:03:10.4634270Z Running tests...
2023-01-11T22:03:10.4634467Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4634732Z   test_wait_all_workers_twice_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32390
2023-01-11T22:03:10.4634891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32391
2023-01-11T22:03:10.4635036Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32392
2023-01-11T22:03:10.4635190Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32393
2023-01-11T22:03:10.4635467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4635590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4635870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4636004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4636192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwkietfe9
2023-01-11T22:03:10.4636387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwkietfe9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4636659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4636773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4637052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4637185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4637370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaf0fpiqq
2023-01-11T22:03:10.4637563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaf0fpiqq/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4637838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4637961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4638278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4638431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4638678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3afa2nxr
2023-01-11T22:03:10.4639004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3afa2nxr/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4639288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4639413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4639696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4639830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4640020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyfc5fobt
2023-01-11T22:03:10.4640220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyfc5fobt/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4640373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4640538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4640705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4640868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4641048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:03:10.4641226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:03:10.4641403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:03:10.4641581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:03:10.4641888Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4642176Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4642474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4642764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:03:10.4643034Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.4643307Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4643571Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2023-01-11T22:03:10.4643830Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.4644085Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.4644344Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:10.4644460Z ok (2.942s)
2023-01-11T22:03:10.4644466Z 
2023-01-11T22:03:10.4644707Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4644771Z Ran 1 test in 2.943s
2023-01-11T22:03:10.4644777Z 
2023-01-11T22:03:10.4644837Z OK
2023-01-11T22:03:10.4644842Z 
2023-01-11T22:03:10.4644930Z Generating XML reports...
2023-01-11T22:03:10.4645325Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220248.xml
2023-01-11T22:03:10.4645600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4645724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4646009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4646149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4646333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperxwmkb3
2023-01-11T22:03:10.4646518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperxwmkb3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4646775Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4646780Z 
2023-01-11T22:03:10.4646852Z Running tests...
2023-01-11T22:03:10.4647048Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4647305Z   test_world_size_one_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32637
2023-01-11T22:03:10.4647461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32638
2023-01-11T22:03:10.4647615Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32639
2023-01-11T22:03:10.4647771Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32640
2023-01-11T22:03:10.4648037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4648159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4648441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4648576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4648760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8c5gsh6y
2023-01-11T22:03:10.4648953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8c5gsh6y/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4649225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4649350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4649629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4649753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4649937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74r_ap_d
2023-01-11T22:03:10.4650131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74r_ap_d/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4650405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4650528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4650808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4650942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4651157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlg7fqu3
2023-01-11T22:03:10.4651366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlg7fqu3/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4651637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4651760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4652038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4652171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4652352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp488t3kr5
2023-01-11T22:03:10.4652542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp488t3kr5/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4652707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4652874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4653023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4653182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:03:10.4653248Z ok (2.447s)
2023-01-11T22:03:10.4653253Z 
2023-01-11T22:03:10.4653449Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4653524Z Ran 1 test in 2.447s
2023-01-11T22:03:10.4653529Z 
2023-01-11T22:03:10.4653589Z OK
2023-01-11T22:03:10.4653594Z 
2023-01-11T22:03:10.4653676Z Generating XML reports...
2023-01-11T22:03:10.4654065Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220253.xml
2023-01-11T22:03:10.4654340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4654454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4654734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4654869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4655054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2we1p4r2
2023-01-11T22:03:10.4655250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2we1p4r2/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4655505Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4655510Z 
2023-01-11T22:03:10.4655584Z Running tests...
2023-01-11T22:03:10.4655782Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4656057Z   test_create_remote_module_from_module_rref (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32746
2023-01-11T22:03:10.4656216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32747
2023-01-11T22:03:10.4656374Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32748
2023-01-11T22:03:10.4656652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4656776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4657059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4657195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4657381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5re_p38
2023-01-11T22:03:10.4657686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4657825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4658106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4658245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4658428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ug6dp5z
2023-01-11T22:03:10.4658621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5re_p38/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4658813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ug6dp5z/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4659085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4659212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4659480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4659618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4659804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgtx19t7
2023-01-11T22:03:10.4660001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgtx19t7/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4660166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4660333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4660497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4660565Z ok (2.539s)
2023-01-11T22:03:10.4660573Z 
2023-01-11T22:03:10.4660774Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4660839Z Ran 1 test in 2.539s
2023-01-11T22:03:10.4660844Z 
2023-01-11T22:03:10.4660909Z OK
2023-01-11T22:03:10.4660914Z 
2023-01-11T22:03:10.4660999Z Generating XML reports...
2023-01-11T22:03:10.4661414Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20230111220257.xml
2023-01-11T22:03:10.4661789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4661917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4662200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4662337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4662510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8o1_csa
2023-01-11T22:03:10.4662709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8o1_csa/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4662968Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4662974Z 
2023-01-11T22:03:10.4663049Z Running tests...
2023-01-11T22:03:10.4663248Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4663527Z   test_send_remote_module_over_the_wire (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 448
2023-01-11T22:03:10.4663684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 449
2023-01-11T22:03:10.4663837Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 450
2023-01-11T22:03:10.4664114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4664279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4664599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4664739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4664926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd409ocib
2023-01-11T22:03:10.4665122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd409ocib/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4665396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4665521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4665806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4665944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4666119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8mgusp9h
2023-01-11T22:03:10.4666316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8mgusp9h/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4666591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4666715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4667000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4667136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4667322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rrsui71
2023-01-11T22:03:10.4667517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rrsui71/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4667673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4667843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4668008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4668078Z ok (2.547s)
2023-01-11T22:03:10.4668083Z 
2023-01-11T22:03:10.4668282Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4668358Z Ran 1 test in 2.548s
2023-01-11T22:03:10.4668363Z 
2023-01-11T22:03:10.4668425Z OK
2023-01-11T22:03:10.4668430Z 
2023-01-11T22:03:10.4668515Z Generating XML reports...
2023-01-11T22:03:10.4668930Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20230111220302.xml
2023-01-11T22:03:10.4669195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4669323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4669614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4669748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4669934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm2vs1sj9
2023-01-11T22:03:10.4670128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm2vs1sj9/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4670384Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2023-01-11T22:03:10.4670389Z 
2023-01-11T22:03:10.4670462Z Running tests...
2023-01-11T22:03:10.4670648Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4670984Z   test_send_remote_module_over_the_wire_script_not_supported (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 618
2023-01-11T22:03:10.4671168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 619
2023-01-11T22:03:10.4671325Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 620
2023-01-11T22:03:10.4671600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4671723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4672007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4672144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4672329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6cd26m_m
2023-01-11T22:03:10.4672516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6cd26m_m/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4672791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4672914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4673198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4673331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4673515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxktywpkk
2023-01-11T22:03:10.4673711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxktywpkk/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4673983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:03:10.4674109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:03:10.4674378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:03:10.4674511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:03:10.4674692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bgux_g4
2023-01-11T22:03:10.4674882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bgux_g4/_remote_module_non_scriptable.py
2023-01-11T22:03:10.4675046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:03:10.4675208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:03:10.4675368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:03:10.4675643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6cd26m_m/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.4675920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxktywpkk/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.4676187Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmp6cd26m_m/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T22:03:10.4676254Z ok (2.542s)
2023-01-11T22:03:10.4676259Z 
2023-01-11T22:03:10.4676457Z ----------------------------------------------------------------------
2023-01-11T22:03:10.4676533Z Ran 1 test in 2.543s
2023-01-11T22:03:10.4676539Z 
2023-01-11T22:03:10.4676599Z OK
2023-01-11T22:03:10.4676604Z 
2023-01-11T22:03:10.4676686Z Generating XML reports...
2023-01-11T22:03:10.4677097Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20230111220306.xml
2023-01-11T22:03:10.4677135Z 
2023-01-11T22:03:10.4677512Z ##[endgroup]
2023-01-11T22:03:10.4677938Z FINISHED PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_g_gm984m)
2023-01-11T22:03:10.4677959Z 
2023-01-11T22:03:10.4678001Z 
2023-01-11T22:03:10.4678060Z real	77m2.126s
2023-01-11T22:03:10.4678129Z user	150m6.199s
2023-01-11T22:03:10.4678198Z sys	33m35.838s
2023-01-11T22:03:10.4678276Z + assert_git_not_dirty
2023-01-11T22:03:10.4678424Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]]
2023-01-11T22:03:10.4678564Z + [[ linux-focal-py3.7-gcc7 != *xla* ]]
2023-01-11T22:03:10.4678677Z ++ git status --porcelain
2023-01-11T22:03:13.7010597Z + git_status=
2023-01-11T22:03:13.7010963Z + [[ -n '' ]]
2023-01-11T22:03:13.7011221Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]]
2023-01-11T22:03:13.7011426Z + [[ 1 == 1 ]]
2023-01-11T22:03:13.7011588Z + test_rpc
2023-01-11T22:03:13.7011835Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]]
2023-01-11T22:03:13.7012091Z + echo 'Testing RPC C++ tests'
2023-01-11T22:03:13.7012286Z Testing RPC C++ tests
2023-01-11T22:03:13.7013111Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.7/site-packages/torch/bin
2023-01-11T22:03:13.7047873Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.7/site-packages/torch/bin
2023-01-11T22:03:13.7055986Z + ln -sf '/opt/conda/lib/python3.7/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.7/site-packages/torch/bin
2023-01-11T22:03:13.7063368Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc
2023-01-11T22:03:13.7063931Z + mkdir -p test/test-reports/cpp-rpc/test_rpc
2023-01-11T22:03:13.7121963Z + /opt/conda/lib/python3.7/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml
2023-01-11T22:03:13.9776472Z CUDA not available. Disabling CUDA and MultiCUDA tests
2023-01-11T22:03:13.9777251Z Note: Google Test filter = *-*_CUDA:*_MultiCUDA
2023-01-11T22:03:13.9777572Z [==========] Running 8 tests from 3 test suites.
2023-01-11T22:03:13.9777890Z [----------] Global test environment set-up.
2023-01-11T22:03:13.9778175Z [----------] 4 tests from WireSerialize
2023-01-11T22:03:13.9778450Z [ RUN      ] WireSerialize.Base
2023-01-11T22:03:13.9951625Z [       OK ] WireSerialize.Base (17 ms)
2023-01-11T22:03:13.9952104Z [ RUN      ] WireSerialize.RecopySparseTensors
2023-01-11T22:03:14.0029801Z [       OK ] WireSerialize.RecopySparseTensors (7 ms)
2023-01-11T22:03:14.0030194Z [ RUN      ] WireSerialize.CloneSparseTensors
2023-01-11T22:03:14.0100034Z [       OK ] WireSerialize.CloneSparseTensors (7 ms)
2023-01-11T22:03:14.0100351Z [ RUN      ] WireSerialize.Errors
2023-01-11T22:03:14.0124788Z [       OK ] WireSerialize.Errors (2 ms)
2023-01-11T22:03:14.0125405Z [----------] 4 tests from WireSerialize (34 ms total)
2023-01-11T22:03:14.0125663Z 
2023-01-11T22:03:14.0125832Z [----------] 1 test from TestE2ETensorPipe
2023-01-11T22:03:14.0126145Z [ RUN      ] TestE2ETensorPipe.TestTrainingLoop
2023-01-11T22:03:14.4067533Z [W tensorpipe_agent.cpp:725] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T22:03:14.4081314Z [       OK ] TestE2ETensorPipe.TestTrainingLoop (395 ms)
2023-01-11T22:03:14.4082012Z [----------] 1 test from TestE2ETensorPipe (395 ms total)
2023-01-11T22:03:14.4082338Z 
2023-01-11T22:03:14.4082628Z [----------] 3 tests from TensorpipeSerialize
2023-01-11T22:03:14.4083154Z [ RUN      ] TensorpipeSerialize.Base
2023-01-11T22:03:14.4083462Z [       OK ] TensorpipeSerialize.Base (0 ms)
2023-01-11T22:03:14.4083869Z [ RUN      ] TensorpipeSerialize.RecopySparseTensors
2023-01-11T22:03:14.4149306Z [       OK ] TensorpipeSerialize.RecopySparseTensors (6 ms)
2023-01-11T22:03:14.4150044Z [ RUN      ] TensorpipeSerialize.NoDeleterTensors
2023-01-11T22:03:14.4150697Z [       OK ] TensorpipeSerialize.NoDeleterTensors (0 ms)
2023-01-11T22:03:14.4151300Z [----------] 3 tests from TensorpipeSerialize (6 ms total)
2023-01-11T22:03:14.4151517Z 
2023-01-11T22:03:14.4151677Z [----------] Global test environment tear-down
2023-01-11T22:03:14.4153909Z [==========] 8 tests from 3 test suites ran. (437 ms total)
2023-01-11T22:03:14.4154492Z [  PASSED  ] 8 tests.
2023-01-11T22:03:14.4154797Z 
2023-01-11T22:03:14.4155050Z   YOU HAVE 1 DISABLED TEST
2023-01-11T22:03:14.4155292Z 
2023-01-11T22:03:14.4959879Z 
2023-01-11T22:03:14.5037665Z ##[group]Run cat test/**/*.log || true
2023-01-11T22:03:14.5037897Z cat test/**/*.log || true
2023-01-11T22:03:14.6019595Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:03:14.6019832Z env:
2023-01-11T22:03:14.6020015Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:03:14.6020278Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:03:14.6020541Z ##[endgroup]
2023-01-11T22:03:14.6061330Z cat: test/**/*.log: No such file or directory
2023-01-11T22:03:14.6093670Z Prepare all required actions
2023-01-11T22:03:14.6093967Z Getting action download info
2023-01-11T22:03:14.7789649Z ##[group]Run ./.github/actions/get-workflow-job-id
2023-01-11T22:03:14.7789858Z with:
2023-01-11T22:03:14.7790194Z   github-token: ***
2023-01-11T22:03:14.7790359Z env:
2023-01-11T22:03:14.7790515Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:03:14.7790788Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:03:14.7791163Z ##[endgroup]
2023-01-11T22:03:14.7818047Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-01-11T22:03:14.7818283Z with:
2023-01-11T22:03:14.7818444Z   shell: bash
2023-01-11T22:03:14.7818605Z   timeout_minutes: 10
2023-01-11T22:03:14.7818786Z   max_attempts: 5
2023-01-11T22:03:14.7818964Z   retry_wait_seconds: 30
2023-01-11T22:03:14.7819368Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}"

2023-01-11T22:03:14.7819747Z   polling_interval_seconds: 1
2023-01-11T22:03:14.7819930Z   warning_on_retry: true
2023-01-11T22:03:14.7820115Z   continue_on_error: false
2023-01-11T22:03:14.7820287Z env:
2023-01-11T22:03:14.7820443Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:03:14.7820710Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:03:14.7821095Z   GITHUB_TOKEN: ***
2023-01-11T22:03:14.7821266Z ##[endgroup]
2023-01-11T22:03:14.8342076Z + python3 -m pip install requests==2.26.0
2023-01-11T22:03:15.6918374Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T22:03:15.7206640Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0)
2023-01-11T22:03:15.7363064Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.12.7)
2023-01-11T22:03:15.7378195Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.13)
2023-01-11T22:03:15.7539873Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4)
2023-01-11T22:03:15.7557518Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12)
2023-01-11T22:03:15.9693526Z ++ python3 .github/scripts/get_workflow_job_id.py 3896099317 i-03e06aa264bd8cb8d
2023-01-11T22:03:19.8178815Z + GHA_WORKFLOW_JOB_ID=10588877413
2023-01-11T22:03:19.8179376Z + echo job-id=10588877413
2023-01-11T22:03:19.8352899Z Command completed after 1 attempt(s).
2023-01-11T22:03:19.8455021Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2023-01-11T22:03:19.8455260Z kill "$MONITOR_SCRIPT_PID"
2023-01-11T22:03:19.8466544Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:03:19.8466770Z env:
2023-01-11T22:03:19.8466949Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:03:19.8467208Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:03:19.8467475Z   MONITOR_SCRIPT_PID: 30581
2023-01-11T22:03:19.8467658Z ##[endgroup]
2023-01-11T22:03:19.8553806Z Prepare all required actions
2023-01-11T22:03:19.8554060Z Getting action download info
2023-01-11T22:03:20.0155077Z Download action repository 'actions/upload-artifact@v3' (SHA:0b7f8abb1508181956e8e162db84b466c27e18ce)
2023-01-11T22:03:20.1879996Z ##[group]Run ./.github/actions/upload-test-artifacts
2023-01-11T22:03:20.1880208Z with:
2023-01-11T22:03:20.1880443Z   file-suffix: test-distributed-1-2-linux.2xlarge_10588877413
2023-01-11T22:03:20.1880674Z env:
2023-01-11T22:03:20.1880832Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:03:20.1881107Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:03:20.1881362Z ##[endgroup]
2023-01-11T22:03:20.1902741Z ##[group]Run # Remove any previous test jsons if they exist
2023-01-11T22:03:20.1903008Z # Remove any previous test jsons if they exist
2023-01-11T22:03:20.1903236Z rm -f test-jsons-*.zip
2023-01-11T22:03:20.1903520Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2023-01-11T22:03:20.1914966Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:03:20.1915189Z env:
2023-01-11T22:03:20.1915367Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:03:20.1915625Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:03:20.1915948Z   FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_10588877413
2023-01-11T22:03:20.1916186Z ##[endgroup]
2023-01-11T22:03:20.2058140Z   adding: test/allowlist_for_publicAPI.json (deflated 78%)
2023-01-11T22:03:20.2085002Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2023-01-11T22:03:20.2090880Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2023-01-11T22:03:20.2092094Z   adding: test/.pytorch-slow-tests.json (deflated 74%)
2023-01-11T22:03:20.2096499Z   adding: test/.pytorch-disabled-tests.json (deflated 84%)
2023-01-11T22:03:20.2117479Z ##[group]Run # Remove any previous test reports if they exist
2023-01-11T22:03:20.2117765Z # Remove any previous test reports if they exist
2023-01-11T22:03:20.2118009Z rm -f test-reports-*.zip
2023-01-11T22:03:20.2118271Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2023-01-11T22:03:20.2129185Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:03:20.2129400Z env:
2023-01-11T22:03:20.2129572Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:03:20.2129832Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:03:20.2130153Z   FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_10588877413
2023-01-11T22:03:20.2130386Z ##[endgroup]
2023-01-11T22:03:20.2289640Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_2d_model_state_checkpoint/TEST-Test2dModelStateCheckpoint-20230111204613.xml (deflated 46%)
2023-01-11T22:03:20.2290929Z   adding: test/test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshCollectiveTest-20230111204617.xml (deflated 84%)
2023-01-11T22:03:20.2292196Z   adding: test/test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshTest-20230111204617.xml (deflated 68%)
2023-01-11T22:03:20.2293250Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_nested_dict/TEST-TestFlattening-20230111211853.xml (deflated 54%)
2023-01-11T22:03:20.2294330Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors/TEST-TestDedupTensor-20230111211855.xml (deflated 41%)
2023-01-11T22:03:20.2295421Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_traverse/TEST-TestTraverse-20230111211857.xml (deflated 77%)
2023-01-11T22:03:20.2296510Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20230111211858.xml (deflated 44%)
2023-01-11T22:03:20.2297525Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20230111211858.xml (deflated 69%)
2023-01-11T22:03:20.2298661Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_utils/TEST-TestMedatadaIndex-20230111211900.xml (deflated 71%)
2023-01-11T22:03:20.2299739Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20230111211902.xml (deflated 43%)
2023-01-11T22:03:20.2300956Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20230111211904.xml (deflated 46%)
2023-01-11T22:03:20.2302407Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20230111211904.xml (deflated 74%)
2023-01-11T22:03:20.2303760Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20230111211904.xml (deflated 45%)
2023-01-11T22:03:20.2305119Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20230111211906.xml (deflated 44%)
2023-01-11T22:03:20.2306348Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20230111211906.xml (deflated 43%)
2023-01-11T22:03:20.2307594Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20230111211909.xml (deflated 67%)
2023-01-11T22:03:20.2308743Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20230111211909.xml (deflated 79%)
2023-01-11T22:03:20.2309913Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_fsdp_optim_state/TEST-FsdpOptimStateCheckpoint-20230111211913.xml (deflated 43%)
2023-01-11T22:03:20.2311025Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20230111211917.xml (deflated 41%)
2023-01-11T22:03:20.2312158Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20230111211921.xml (deflated 47%)
2023-01-11T22:03:20.2313213Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20230111211924.xml (deflated 59%)
2023-01-11T22:03:20.2314253Z   adding: test/test-reports/python-unittest/distributed.test_c10d_error_logger/TEST-C10dErrorLoggerTest-20230111211930.xml (deflated 52%)
2023-01-11T22:03:20.2315287Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20230111211935.xml (deflated 59%)
2023-01-11T22:03:20.2316409Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20230111211940.xml (deflated 71%)
2023-01-11T22:03:20.2317524Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20230111211940.xml (deflated 69%)
2023-01-11T22:03:20.2318727Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20230111211940.xml (deflated 66%)
2023-01-11T22:03:20.2320132Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20230111211946.xml (deflated 44%)
2023-01-11T22:03:20.2321372Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20230111211946.xml (deflated 44%)
2023-01-11T22:03:20.2322544Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20230111211946.xml (deflated 61%)
2023-01-11T22:03:20.2323670Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20230111211946.xml (deflated 59%)
2023-01-11T22:03:20.2324776Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20230111211946.xml (deflated 60%)
2023-01-11T22:03:20.2326002Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20230111211946.xml (deflated 62%)
2023-01-11T22:03:20.2327131Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20230111211946.xml (deflated 62%)
2023-01-11T22:03:20.2328325Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20230111211946.xml (deflated 90%)
2023-01-11T22:03:20.2329309Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20230111211946.xml (deflated 70%)
2023-01-11T22:03:20.2330500Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20230111211946.xml (deflated 88%)
2023-01-11T22:03:20.2331624Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20230111211946.xml (deflated 86%)
2023-01-11T22:03:20.2332551Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20230111211946.xml (deflated 63%)
2023-01-11T22:03:20.2333692Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20230111211951.xml (deflated 64%)
2023-01-11T22:03:20.2334368Z   adding: test/test-reports/python-unittest/distributed._tensor.test_math_ops/TEST-DistMathOpsTest-20230111211958.xml (deflated 59%)
2023-01-11T22:03:20.2335395Z   adding: test/test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20230111212005.xml (deflated 69%)
2023-01-11T22:03:20.2336012Z   adding: test/test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateStateDictTest-20230111212014.xml (deflated 59%)
2023-01-11T22:03:20.2336818Z   adding: test/test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateTest-20230111212014.xml (deflated 63%)
2023-01-11T22:03:20.2337943Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20230111212025.xml (deflated 80%)
2023-01-11T22:03:20.2338951Z   adding: test/test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops/TEST-TPShardingOpsTest-20230111212036.xml (deflated 75%)
2023-01-11T22:03:20.2340082Z   adding: test/test-reports/python-unittest/distributed.tensor.parallel.test_parallelize_api/TEST-TensorParallelAPITests-20230111212049.xml (deflated 76%)
2023-01-11T22:03:20.2341137Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20230111212103.xml (deflated 38%)
2023-01-11T22:03:20.2342251Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111212106.xml (deflated 42%)
2023-01-11T22:03:20.2343389Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111212108.xml (deflated 41%)
2023-01-11T22:03:20.2344648Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111212110.xml (deflated 41%)
2023-01-11T22:03:20.2345795Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111212112.xml (deflated 43%)
2023-01-11T22:03:20.2346989Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111212114.xml (deflated 42%)
2023-01-11T22:03:20.2348183Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111212117.xml (deflated 42%)
2023-01-11T22:03:20.2349379Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111212121.xml (deflated 42%)
2023-01-11T22:03:20.2350635Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111212125.xml (deflated 42%)
2023-01-11T22:03:20.2351733Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111212129.xml (deflated 40%)
2023-01-11T22:03:20.2352707Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111212131.xml (deflated 40%)
2023-01-11T22:03:20.2353681Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111212133.xml (deflated 40%)
2023-01-11T22:03:20.2354648Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111212135.xml (deflated 40%)
2023-01-11T22:03:20.2355689Z   adding: test/test-reports/python-unittest/distributed._composable.test_compose/TEST-TestFSDPCheckpoint-20230111212137.xml (deflated 79%)
2023-01-11T22:03:20.2356771Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params/TEST-TestFlattenParams-20230111212152.xml (deflated 78%)
2023-01-11T22:03:20.2357922Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20230111212209.xml (deflated 69%)
2023-01-11T22:03:20.2359243Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20230111212209.xml (deflated 85%)
2023-01-11T22:03:20.2360451Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20230111212226.xml (deflated 84%)
2023-01-11T22:03:20.2361686Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111212244.xml (deflated 44%)
2023-01-11T22:03:20.2363019Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111212245.xml (deflated 44%)
2023-01-11T22:03:20.2364354Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111212247.xml (deflated 44%)
2023-01-11T22:03:20.2365616Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20230111212249.xml (deflated 41%)
2023-01-11T22:03:20.2366787Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20230111212251.xml (deflated 42%)
2023-01-11T22:03:20.2367944Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20230111212253.xml (deflated 42%)
2023-01-11T22:03:20.2369121Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20230111212254.xml (deflated 42%)
2023-01-11T22:03:20.2370316Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212256.xml (deflated 43%)
2023-01-11T22:03:20.2371550Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212300.xml (deflated 44%)
2023-01-11T22:03:20.2372780Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212304.xml (deflated 43%)
2023-01-11T22:03:20.2373980Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212308.xml (deflated 44%)
2023-01-11T22:03:20.2375164Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212311.xml (deflated 44%)
2023-01-11T22:03:20.2376355Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212315.xml (deflated 43%)
2023-01-11T22:03:20.2377580Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212319.xml (deflated 43%)
2023-01-11T22:03:20.2378784Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111212323.xml (deflated 43%)
2023-01-11T22:03:20.2380051Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20230111212327.xml (deflated 81%)
2023-01-11T22:03:20.2381397Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20230111212327.xml (deflated 55%)
2023-01-11T22:03:20.2383082Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20230111212327.xml (deflated 61%)
2023-01-11T22:03:20.2384423Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212348.xml (deflated 41%)
2023-01-11T22:03:20.2385595Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212351.xml (deflated 41%)
2023-01-11T22:03:20.2386770Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212355.xml (deflated 41%)
2023-01-11T22:03:20.2387932Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212359.xml (deflated 41%)
2023-01-11T22:03:20.2389083Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212402.xml (deflated 41%)
2023-01-11T22:03:20.2390240Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212406.xml (deflated 41%)
2023-01-11T22:03:20.2391390Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212410.xml (deflated 42%)
2023-01-11T22:03:20.2392552Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212414.xml (deflated 41%)
2023-01-11T22:03:20.2393685Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111212417.xml (deflated 41%)
2023-01-11T22:03:20.2394835Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111212421.xml (deflated 41%)
2023-01-11T22:03:20.2395974Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111212423.xml (deflated 41%)
2023-01-11T22:03:20.2397120Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111212424.xml (deflated 41%)
2023-01-11T22:03:20.2398234Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111212426.xml (deflated 41%)
2023-01-11T22:03:20.2399475Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111212427.xml (deflated 41%)
2023-01-11T22:03:20.2400763Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20230111212429.xml (deflated 56%)
2023-01-11T22:03:20.2401996Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20230111212429.xml (deflated 80%)
2023-01-11T22:03:20.2403190Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20230111212429.xml (deflated 79%)
2023-01-11T22:03:20.2404334Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20230111212429.xml (deflated 64%)
2023-01-11T22:03:20.2405402Z   adding: test/test-reports/python-unittest/distributed._tensor.test_common_rules/TEST-CommonRulesTest-20230111212451.xml (deflated 80%)
2023-01-11T22:03:20.2406580Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20230111212514.xml (deflated 92%)
2023-01-11T22:03:20.2407652Z   adding: test/test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20230111212543.xml (deflated 70%)
2023-01-11T22:03:20.2408639Z   adding: test/test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20230111212543.xml (deflated 75%)
2023-01-11T22:03:20.2409636Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20230111212613.xml (deflated 81%)
2023-01-11T22:03:20.2410708Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMiscWorldSize1-20230111212613.xml (deflated 42%)
2023-01-11T22:03:20.2412177Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20230111212645.xml (deflated 92%)
2023-01-11T22:03:20.2413555Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20230111212645.xml (deflated 73%)
2023-01-11T22:03:20.2414768Z   adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20230111212732.xml (deflated 86%)
2023-01-11T22:03:20.2415831Z   adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20230111212732.xml (deflated 86%)
2023-01-11T22:03:20.2416986Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20230111212834.xml (deflated 42%)
2023-01-11T22:03:20.2418188Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20230111212836.xml (deflated 42%)
2023-01-11T22:03:20.2419383Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20230111212838.xml (deflated 41%)
2023-01-11T22:03:20.2420565Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20230111212843.xml (deflated 42%)
2023-01-11T22:03:20.2421932Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20230111212847.xml (deflated 42%)
2023-01-11T22:03:20.2423246Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20230111212853.xml (deflated 42%)
2023-01-11T22:03:20.2424561Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20230111212900.xml (deflated 42%)
2023-01-11T22:03:20.2425865Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20230111212906.xml (deflated 41%)
2023-01-11T22:03:20.2427113Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212912.xml (deflated 43%)
2023-01-11T22:03:20.2428410Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212916.xml (deflated 41%)
2023-01-11T22:03:20.2429634Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212921.xml (deflated 41%)
2023-01-11T22:03:20.2430835Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212926.xml (deflated 41%)
2023-01-11T22:03:20.2432019Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212931.xml (deflated 40%)
2023-01-11T22:03:20.2433200Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212935.xml (deflated 40%)
2023-01-11T22:03:20.2434445Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212940.xml (deflated 41%)
2023-01-11T22:03:20.2435643Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212944.xml (deflated 40%)
2023-01-11T22:03:20.2436813Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212949.xml (deflated 42%)
2023-01-11T22:03:20.2438001Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212951.xml (deflated 40%)
2023-01-11T22:03:20.2439243Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212956.xml (deflated 42%)
2023-01-11T22:03:20.2440435Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111212958.xml (deflated 41%)
2023-01-11T22:03:20.2441608Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213003.xml (deflated 41%)
2023-01-11T22:03:20.2442787Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213008.xml (deflated 41%)
2023-01-11T22:03:20.2443947Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213012.xml (deflated 41%)
2023-01-11T22:03:20.2445140Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213017.xml (deflated 41%)
2023-01-11T22:03:20.2446324Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213021.xml (deflated 40%)
2023-01-11T22:03:20.2447487Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213026.xml (deflated 41%)
2023-01-11T22:03:20.2448671Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213031.xml (deflated 41%)
2023-01-11T22:03:20.2449865Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213035.xml (deflated 40%)
2023-01-11T22:03:20.2451050Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213040.xml (deflated 41%)
2023-01-11T22:03:20.2452199Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213045.xml (deflated 41%)
2023-01-11T22:03:20.2453386Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213049.xml (deflated 41%)
2023-01-11T22:03:20.2454581Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213104.xml (deflated 42%)
2023-01-11T22:03:20.2455770Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213106.xml (deflated 40%)
2023-01-11T22:03:20.2457012Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213111.xml (deflated 41%)
2023-01-11T22:03:20.2458202Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213115.xml (deflated 40%)
2023-01-11T22:03:20.2459407Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213120.xml (deflated 41%)
2023-01-11T22:03:20.2460605Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213125.xml (deflated 49%)
2023-01-11T22:03:20.2461953Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213129.xml (deflated 41%)
2023-01-11T22:03:20.2463163Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213134.xml (deflated 42%)
2023-01-11T22:03:20.2464351Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213139.xml (deflated 42%)
2023-01-11T22:03:20.2465532Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213144.xml (deflated 42%)
2023-01-11T22:03:20.2466713Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213149.xml (deflated 41%)
2023-01-11T22:03:20.2467881Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213153.xml (deflated 42%)
2023-01-11T22:03:20.2469095Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213156.xml (deflated 41%)
2023-01-11T22:03:20.2470287Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213200.xml (deflated 41%)
2023-01-11T22:03:20.2471471Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213205.xml (deflated 41%)
2023-01-11T22:03:20.2472647Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213210.xml (deflated 42%)
2023-01-11T22:03:20.2473832Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213214.xml (deflated 41%)
2023-01-11T22:03:20.2475013Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213219.xml (deflated 41%)
2023-01-11T22:03:20.2476207Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213224.xml (deflated 41%)
2023-01-11T22:03:20.2477394Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213229.xml (deflated 41%)
2023-01-11T22:03:20.2478616Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213234.xml (deflated 41%)
2023-01-11T22:03:20.2479925Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213239.xml (deflated 41%)
2023-01-11T22:03:20.2481116Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213243.xml (deflated 41%)
2023-01-11T22:03:20.2482292Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213248.xml (deflated 41%)
2023-01-11T22:03:20.2483498Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213253.xml (deflated 41%)
2023-01-11T22:03:20.2484763Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213258.xml (deflated 41%)
2023-01-11T22:03:20.2485958Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213303.xml (deflated 41%)
2023-01-11T22:03:20.2487127Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213308.xml (deflated 41%)
2023-01-11T22:03:20.2488301Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213312.xml (deflated 41%)
2023-01-11T22:03:20.2489504Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213317.xml (deflated 44%)
2023-01-11T22:03:20.2490744Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213322.xml (deflated 41%)
2023-01-11T22:03:20.2491958Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213326.xml (deflated 42%)
2023-01-11T22:03:20.2493154Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213331.xml (deflated 42%)
2023-01-11T22:03:20.2494353Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213336.xml (deflated 41%)
2023-01-11T22:03:20.2495546Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213340.xml (deflated 41%)
2023-01-11T22:03:20.2496753Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213345.xml (deflated 41%)
2023-01-11T22:03:20.2497944Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213349.xml (deflated 40%)
2023-01-11T22:03:20.2499141Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213355.xml (deflated 41%)
2023-01-11T22:03:20.2500336Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213400.xml (deflated 41%)
2023-01-11T22:03:20.2501516Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20230111213405.xml (deflated 41%)
2023-01-11T22:03:20.2502673Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20230111213409.xml (deflated 42%)
2023-01-11T22:03:20.2503858Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20230111213411.xml (deflated 42%)
2023-01-11T22:03:20.2505030Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20230111213416.xml (deflated 42%)
2023-01-11T22:03:20.2506201Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20230111213421.xml (deflated 42%)
2023-01-11T22:03:20.2507385Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20230111213423.xml (deflated 41%)
2023-01-11T22:03:20.2508566Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20230111213428.xml (deflated 41%)
2023-01-11T22:03:20.2530029Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20230111213432.xml (deflated 41%)
2023-01-11T22:03:20.2531332Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20230111213437.xml (deflated 41%)
2023-01-11T22:03:20.2532625Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213442.xml (deflated 40%)
2023-01-11T22:03:20.2533711Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213446.xml (deflated 39%)
2023-01-11T22:03:20.2534786Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213451.xml (deflated 40%)
2023-01-11T22:03:20.2535882Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213456.xml (deflated 41%)
2023-01-11T22:03:20.2536985Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213501.xml (deflated 40%)
2023-01-11T22:03:20.2537933Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213505.xml (deflated 40%)
2023-01-11T22:03:20.2538552Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213510.xml (deflated 40%)
2023-01-11T22:03:20.2539157Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213515.xml (deflated 40%)
2023-01-11T22:03:20.2539765Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213519.xml (deflated 40%)
2023-01-11T22:03:20.2540356Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213524.xml (deflated 40%)
2023-01-11T22:03:20.2540960Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213528.xml (deflated 40%)
2023-01-11T22:03:20.2541560Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213533.xml (deflated 40%)
2023-01-11T22:03:20.2542402Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213538.xml (deflated 40%)
2023-01-11T22:03:20.2543001Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213542.xml (deflated 40%)
2023-01-11T22:03:20.2543584Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213547.xml (deflated 40%)
2023-01-11T22:03:20.2544188Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213551.xml (deflated 40%)
2023-01-11T22:03:20.2544791Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213556.xml (deflated 40%)
2023-01-11T22:03:20.2545397Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213601.xml (deflated 40%)
2023-01-11T22:03:20.2545991Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213605.xml (deflated 40%)
2023-01-11T22:03:20.2546586Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213610.xml (deflated 40%)
2023-01-11T22:03:20.2547181Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213615.xml (deflated 41%)
2023-01-11T22:03:20.2547771Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213619.xml (deflated 40%)
2023-01-11T22:03:20.2548352Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213624.xml (deflated 40%)
2023-01-11T22:03:20.2548957Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213629.xml (deflated 41%)
2023-01-11T22:03:20.2549635Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213631.xml (deflated 40%)
2023-01-11T22:03:20.2550236Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213635.xml (deflated 40%)
2023-01-11T22:03:20.2550817Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213640.xml (deflated 40%)
2023-01-11T22:03:20.2551414Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213645.xml (deflated 39%)
2023-01-11T22:03:20.2552011Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213650.xml (deflated 41%)
2023-01-11T22:03:20.2552656Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213654.xml (deflated 40%)
2023-01-11T22:03:20.2553243Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213659.xml (deflated 40%)
2023-01-11T22:03:20.2553836Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213703.xml (deflated 39%)
2023-01-11T22:03:20.2554427Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213708.xml (deflated 39%)
2023-01-11T22:03:20.2555024Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213713.xml (deflated 40%)
2023-01-11T22:03:20.2555606Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213718.xml (deflated 40%)
2023-01-11T22:03:20.2556209Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213722.xml (deflated 40%)
2023-01-11T22:03:20.2556808Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213727.xml (deflated 40%)
2023-01-11T22:03:20.2557406Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213732.xml (deflated 41%)
2023-01-11T22:03:20.2557999Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213736.xml (deflated 40%)
2023-01-11T22:03:20.2558580Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213741.xml (deflated 40%)
2023-01-11T22:03:20.2559248Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213746.xml (deflated 40%)
2023-01-11T22:03:20.2559848Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213751.xml (deflated 40%)
2023-01-11T22:03:20.2560447Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213755.xml (deflated 40%)
2023-01-11T22:03:20.2561030Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213800.xml (deflated 40%)
2023-01-11T22:03:20.2561618Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213805.xml (deflated 40%)
2023-01-11T22:03:20.2562219Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213809.xml (deflated 40%)
2023-01-11T22:03:20.2562815Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213814.xml (deflated 40%)
2023-01-11T22:03:20.2563397Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213819.xml (deflated 39%)
2023-01-11T22:03:20.2564029Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213824.xml (deflated 40%)
2023-01-11T22:03:20.2564622Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213828.xml (deflated 40%)
2023-01-11T22:03:20.2565208Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213833.xml (deflated 40%)
2023-01-11T22:03:20.2565787Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213838.xml (deflated 40%)
2023-01-11T22:03:20.2566386Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213842.xml (deflated 40%)
2023-01-11T22:03:20.2567012Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20230111213847.xml (deflated 40%)
2023-01-11T22:03:20.2567655Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeParameterServerTest-20230111213852.xml (deflated 43%)
2023-01-11T22:03:20.2568345Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeReinforcementLearningRpcTest-20230111213856.xml (deflated 43%)
2023-01-11T22:03:20.2569024Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213901.xml (deflated 41%)
2023-01-11T22:03:20.2569667Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213906.xml (deflated 41%)
2023-01-11T22:03:20.2570308Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213910.xml (deflated 41%)
2023-01-11T22:03:20.2570940Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213915.xml (deflated 41%)
2023-01-11T22:03:20.2571572Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213919.xml (deflated 41%)
2023-01-11T22:03:20.2572213Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213924.xml (deflated 41%)
2023-01-11T22:03:20.2572846Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213928.xml (deflated 41%)
2023-01-11T22:03:20.2573482Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213932.xml (deflated 42%)
2023-01-11T22:03:20.2574097Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213937.xml (deflated 41%)
2023-01-11T22:03:20.2574730Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213941.xml (deflated 42%)
2023-01-11T22:03:20.2575371Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213945.xml (deflated 41%)
2023-01-11T22:03:20.2576004Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213950.xml (deflated 41%)
2023-01-11T22:03:20.2576624Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20230111213954.xml (deflated 41%)
2023-01-11T22:03:20.2577229Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111213958.xml (deflated 40%)
2023-01-11T22:03:20.2577816Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214003.xml (deflated 40%)
2023-01-11T22:03:20.2578425Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214008.xml (deflated 40%)
2023-01-11T22:03:20.2578994Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214012.xml (deflated 40%)
2023-01-11T22:03:20.2579587Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214017.xml (deflated 41%)
2023-01-11T22:03:20.2580168Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214031.xml (deflated 40%)
2023-01-11T22:03:20.2580743Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214035.xml (deflated 40%)
2023-01-11T22:03:20.2581343Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214040.xml (deflated 41%)
2023-01-11T22:03:20.2582140Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214044.xml (deflated 40%)
2023-01-11T22:03:20.2582738Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214049.xml (deflated 41%)
2023-01-11T22:03:20.2583320Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214054.xml (deflated 40%)
2023-01-11T22:03:20.2583887Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214058.xml (deflated 40%)
2023-01-11T22:03:20.2584470Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214103.xml (deflated 40%)
2023-01-11T22:03:20.2585055Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214108.xml (deflated 40%)
2023-01-11T22:03:20.2585635Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214112.xml (deflated 40%)
2023-01-11T22:03:20.2586203Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214117.xml (deflated 40%)
2023-01-11T22:03:20.2586796Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214121.xml (deflated 40%)
2023-01-11T22:03:20.2587376Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214126.xml (deflated 40%)
2023-01-11T22:03:20.2587952Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214131.xml (deflated 40%)
2023-01-11T22:03:20.2588523Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214135.xml (deflated 40%)
2023-01-11T22:03:20.2589084Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214140.xml (deflated 40%)
2023-01-11T22:03:20.2589672Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214145.xml (deflated 40%)
2023-01-11T22:03:20.2590246Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214149.xml (deflated 41%)
2023-01-11T22:03:20.2590819Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214152.xml (deflated 40%)
2023-01-11T22:03:20.2591381Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214156.xml (deflated 40%)
2023-01-11T22:03:20.2591965Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214201.xml (deflated 40%)
2023-01-11T22:03:20.2592633Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214205.xml (deflated 40%)
2023-01-11T22:03:20.2593214Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214210.xml (deflated 40%)
2023-01-11T22:03:20.2593778Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214215.xml (deflated 40%)
2023-01-11T22:03:20.2594359Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214219.xml (deflated 40%)
2023-01-11T22:03:20.2594935Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214224.xml (deflated 40%)
2023-01-11T22:03:20.2595510Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214228.xml (deflated 40%)
2023-01-11T22:03:20.2596108Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214234.xml (deflated 40%)
2023-01-11T22:03:20.2596700Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214239.xml (deflated 40%)
2023-01-11T22:03:20.2597283Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214244.xml (deflated 40%)
2023-01-11T22:03:20.2597856Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214249.xml (deflated 40%)
2023-01-11T22:03:20.2598416Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214253.xml (deflated 40%)
2023-01-11T22:03:20.2599148Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214257.xml (deflated 40%)
2023-01-11T22:03:20.2599787Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214302.xml (deflated 42%)
2023-01-11T22:03:20.2600375Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214307.xml (deflated 40%)
2023-01-11T22:03:20.2600934Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214311.xml (deflated 40%)
2023-01-11T22:03:20.2601506Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214316.xml (deflated 40%)
2023-01-11T22:03:20.2602117Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214320.xml (deflated 40%)
2023-01-11T22:03:20.2602693Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214325.xml (deflated 40%)
2023-01-11T22:03:20.2603298Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214330.xml (deflated 40%)
2023-01-11T22:03:20.2603870Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214334.xml (deflated 40%)
2023-01-11T22:03:20.2604451Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214339.xml (deflated 40%)
2023-01-11T22:03:20.2605041Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214344.xml (deflated 39%)
2023-01-11T22:03:20.2605620Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214348.xml (deflated 40%)
2023-01-11T22:03:20.2606204Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214352.xml (deflated 42%)
2023-01-11T22:03:20.2606776Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214357.xml (deflated 40%)
2023-01-11T22:03:20.2607414Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214403.xml (deflated 40%)
2023-01-11T22:03:20.2608006Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214407.xml (deflated 44%)
2023-01-11T22:03:20.2608591Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214414.xml (deflated 40%)
2023-01-11T22:03:20.2609167Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214419.xml (deflated 40%)
2023-01-11T22:03:20.2609757Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214423.xml (deflated 40%)
2023-01-11T22:03:20.2610376Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214428.xml (deflated 40%)
2023-01-11T22:03:20.2610962Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214432.xml (deflated 41%)
2023-01-11T22:03:20.2611531Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214437.xml (deflated 40%)
2023-01-11T22:03:20.2612117Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214441.xml (deflated 40%)
2023-01-11T22:03:20.2612698Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214446.xml (deflated 40%)
2023-01-11T22:03:20.2613274Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214450.xml (deflated 40%)
2023-01-11T22:03:20.2613842Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214455.xml (deflated 40%)
2023-01-11T22:03:20.2614428Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214500.xml (deflated 41%)
2023-01-11T22:03:20.2615008Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214502.xml (deflated 40%)
2023-01-11T22:03:20.2615590Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214507.xml (deflated 41%)
2023-01-11T22:03:20.2616199Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214511.xml (deflated 40%)
2023-01-11T22:03:20.2616811Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214516.xml (deflated 40%)
2023-01-11T22:03:20.2617401Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214521.xml (deflated 40%)
2023-01-11T22:03:20.2618287Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214525.xml (deflated 40%)
2023-01-11T22:03:20.2618870Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214530.xml (deflated 40%)
2023-01-11T22:03:20.2619435Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214535.xml (deflated 41%)
2023-01-11T22:03:20.2620018Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214540.xml (deflated 41%)
2023-01-11T22:03:20.2620602Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214544.xml (deflated 40%)
2023-01-11T22:03:20.2621184Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214548.xml (deflated 40%)
2023-01-11T22:03:20.2621975Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214553.xml (deflated 39%)
2023-01-11T22:03:20.2622573Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214558.xml (deflated 40%)
2023-01-11T22:03:20.2623161Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214603.xml (deflated 40%)
2023-01-11T22:03:20.2623743Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214607.xml (deflated 40%)
2023-01-11T22:03:20.2624307Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214612.xml (deflated 40%)
2023-01-11T22:03:20.2624949Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214616.xml (deflated 40%)
2023-01-11T22:03:20.2625552Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214621.xml (deflated 40%)
2023-01-11T22:03:20.2626136Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214625.xml (deflated 40%)
2023-01-11T22:03:20.2626715Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214630.xml (deflated 40%)
2023-01-11T22:03:20.2627282Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214635.xml (deflated 40%)
2023-01-11T22:03:20.2627871Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214639.xml (deflated 40%)
2023-01-11T22:03:20.2628458Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214644.xml (deflated 40%)
2023-01-11T22:03:20.2629039Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214649.xml (deflated 41%)
2023-01-11T22:03:20.2629602Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214653.xml (deflated 39%)
2023-01-11T22:03:20.2630193Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214658.xml (deflated 40%)
2023-01-11T22:03:20.2630776Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214702.xml (deflated 40%)
2023-01-11T22:03:20.2631356Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214707.xml (deflated 40%)
2023-01-11T22:03:20.2631921Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214712.xml (deflated 40%)
2023-01-11T22:03:20.2632503Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214716.xml (deflated 40%)
2023-01-11T22:03:20.2633086Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214721.xml (deflated 40%)
2023-01-11T22:03:20.2633667Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214725.xml (deflated 40%)
2023-01-11T22:03:20.2634234Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214730.xml (deflated 41%)
2023-01-11T22:03:20.2634821Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214734.xml (deflated 41%)
2023-01-11T22:03:20.2635402Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214739.xml (deflated 40%)
2023-01-11T22:03:20.2635986Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214743.xml (deflated 40%)
2023-01-11T22:03:20.2636594Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214748.xml (deflated 40%)
2023-01-11T22:03:20.2637166Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214753.xml (deflated 40%)
2023-01-11T22:03:20.2637752Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214757.xml (deflated 40%)
2023-01-11T22:03:20.2638328Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214802.xml (deflated 40%)
2023-01-11T22:03:20.2638963Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214809.xml (deflated 39%)
2023-01-11T22:03:20.2639591Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214817.xml (deflated 40%)
2023-01-11T22:03:20.2640183Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214822.xml (deflated 40%)
2023-01-11T22:03:20.2640766Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214826.xml (deflated 40%)
2023-01-11T22:03:20.2641333Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214831.xml (deflated 40%)
2023-01-11T22:03:20.2641912Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214835.xml (deflated 40%)
2023-01-11T22:03:20.2642494Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214843.xml (deflated 40%)
2023-01-11T22:03:20.2643075Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214851.xml (deflated 40%)
2023-01-11T22:03:20.2643656Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214855.xml (deflated 40%)
2023-01-11T22:03:20.2644221Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214900.xml (deflated 40%)
2023-01-11T22:03:20.2644805Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214904.xml (deflated 40%)
2023-01-11T22:03:20.2645386Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214909.xml (deflated 40%)
2023-01-11T22:03:20.2645966Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214914.xml (deflated 40%)
2023-01-11T22:03:20.2646537Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214918.xml (deflated 40%)
2023-01-11T22:03:20.2647117Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214923.xml (deflated 40%)
2023-01-11T22:03:20.2647690Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214927.xml (deflated 40%)
2023-01-11T22:03:20.2648265Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214934.xml (deflated 39%)
2023-01-11T22:03:20.2648832Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214940.xml (deflated 40%)
2023-01-11T22:03:20.2649418Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214945.xml (deflated 40%)
2023-01-11T22:03:20.2650005Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214950.xml (deflated 40%)
2023-01-11T22:03:20.2650624Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214954.xml (deflated 40%)
2023-01-11T22:03:20.2651190Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111214959.xml (deflated 40%)
2023-01-11T22:03:20.2651775Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215003.xml (deflated 40%)
2023-01-11T22:03:20.2652357Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215008.xml (deflated 40%)
2023-01-11T22:03:20.2652936Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215013.xml (deflated 40%)
2023-01-11T22:03:20.2653544Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215018.xml (deflated 40%)
2023-01-11T22:03:20.2654130Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215022.xml (deflated 40%)
2023-01-11T22:03:20.2654712Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215027.xml (deflated 40%)
2023-01-11T22:03:20.2655294Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215031.xml (deflated 40%)
2023-01-11T22:03:20.2655858Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215036.xml (deflated 40%)
2023-01-11T22:03:20.2656439Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215041.xml (deflated 40%)
2023-01-11T22:03:20.2657031Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215045.xml (deflated 41%)
2023-01-11T22:03:20.2657614Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215050.xml (deflated 41%)
2023-01-11T22:03:20.2658178Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215054.xml (deflated 41%)
2023-01-11T22:03:20.2658756Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215059.xml (deflated 40%)
2023-01-11T22:03:20.2659339Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215104.xml (deflated 40%)
2023-01-11T22:03:20.2659917Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215108.xml (deflated 41%)
2023-01-11T22:03:20.2660480Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215113.xml (deflated 40%)
2023-01-11T22:03:20.2661061Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215117.xml (deflated 40%)
2023-01-11T22:03:20.2661645Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215122.xml (deflated 40%)
2023-01-11T22:03:20.2662415Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215127.xml (deflated 40%)
2023-01-11T22:03:20.2663004Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215131.xml (deflated 40%)
2023-01-11T22:03:20.2663573Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215136.xml (deflated 40%)
2023-01-11T22:03:20.2664162Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215141.xml (deflated 40%)
2023-01-11T22:03:20.2664805Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215145.xml (deflated 40%)
2023-01-11T22:03:20.2665390Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215150.xml (deflated 41%)
2023-01-11T22:03:20.2665965Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215155.xml (deflated 40%)
2023-01-11T22:03:20.2666551Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215159.xml (deflated 40%)
2023-01-11T22:03:20.2667137Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215204.xml (deflated 40%)
2023-01-11T22:03:20.2667768Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215209.xml (deflated 40%)
2023-01-11T22:03:20.2668345Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215214.xml (deflated 40%)
2023-01-11T22:03:20.2668925Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215220.xml (deflated 40%)
2023-01-11T22:03:20.2669504Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215225.xml (deflated 41%)
2023-01-11T22:03:20.2670086Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215229.xml (deflated 40%)
2023-01-11T22:03:20.2670648Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215240.xml (deflated 40%)
2023-01-11T22:03:20.2671231Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215245.xml (deflated 40%)
2023-01-11T22:03:20.2671812Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215250.xml (deflated 40%)
2023-01-11T22:03:20.2672391Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215255.xml (deflated 40%)
2023-01-11T22:03:20.2672954Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215259.xml (deflated 40%)
2023-01-11T22:03:20.2673526Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215304.xml (deflated 40%)
2023-01-11T22:03:20.2674094Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215309.xml (deflated 40%)
2023-01-11T22:03:20.2674669Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215313.xml (deflated 40%)
2023-01-11T22:03:20.2675231Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215318.xml (deflated 41%)
2023-01-11T22:03:20.2675809Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215323.xml (deflated 41%)
2023-01-11T22:03:20.2676391Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215327.xml (deflated 39%)
2023-01-11T22:03:20.2676974Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215332.xml (deflated 40%)
2023-01-11T22:03:20.2677537Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215337.xml (deflated 40%)
2023-01-11T22:03:20.2678117Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215343.xml (deflated 40%)
2023-01-11T22:03:20.2678709Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215348.xml (deflated 40%)
2023-01-11T22:03:20.2679402Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215352.xml (deflated 40%)
2023-01-11T22:03:20.2679971Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215357.xml (deflated 39%)
2023-01-11T22:03:20.2680551Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215402.xml (deflated 40%)
2023-01-11T22:03:20.2681134Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215407.xml (deflated 40%)
2023-01-11T22:03:20.2681718Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215412.xml (deflated 40%)
2023-01-11T22:03:20.2682326Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215416.xml (deflated 40%)
2023-01-11T22:03:20.2682897Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215421.xml (deflated 40%)
2023-01-11T22:03:20.2683483Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215426.xml (deflated 40%)
2023-01-11T22:03:20.2684060Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215430.xml (deflated 41%)
2023-01-11T22:03:20.2684640Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215435.xml (deflated 40%)
2023-01-11T22:03:20.2685202Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215440.xml (deflated 42%)
2023-01-11T22:03:20.2685792Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215445.xml (deflated 41%)
2023-01-11T22:03:20.2686379Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215449.xml (deflated 40%)
2023-01-11T22:03:20.2686956Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215454.xml (deflated 41%)
2023-01-11T22:03:20.2687519Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215458.xml (deflated 40%)
2023-01-11T22:03:20.2688100Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215503.xml (deflated 40%)
2023-01-11T22:03:20.2688684Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215508.xml (deflated 39%)
2023-01-11T22:03:20.2689265Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215512.xml (deflated 40%)
2023-01-11T22:03:20.2689830Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215517.xml (deflated 40%)
2023-01-11T22:03:20.2690407Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215522.xml (deflated 40%)
2023-01-11T22:03:20.2690984Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215527.xml (deflated 41%)
2023-01-11T22:03:20.2691558Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215531.xml (deflated 40%)
2023-01-11T22:03:20.2692116Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215535.xml (deflated 40%)
2023-01-11T22:03:20.2692699Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215540.xml (deflated 40%)
2023-01-11T22:03:20.2693329Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215545.xml (deflated 40%)
2023-01-11T22:03:20.2693911Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215550.xml (deflated 40%)
2023-01-11T22:03:20.2694478Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215554.xml (deflated 40%)
2023-01-11T22:03:20.2695058Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215559.xml (deflated 40%)
2023-01-11T22:03:20.2695638Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215604.xml (deflated 40%)
2023-01-11T22:03:20.2696246Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215608.xml (deflated 40%)
2023-01-11T22:03:20.2696817Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215613.xml (deflated 40%)
2023-01-11T22:03:20.2697392Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215618.xml (deflated 40%)
2023-01-11T22:03:20.2697976Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215622.xml (deflated 40%)
2023-01-11T22:03:20.2698555Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215628.xml (deflated 39%)
2023-01-11T22:03:20.2699124Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215632.xml (deflated 39%)
2023-01-11T22:03:20.2699705Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215637.xml (deflated 40%)
2023-01-11T22:03:20.2700287Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215642.xml (deflated 40%)
2023-01-11T22:03:20.2700862Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215650.xml (deflated 40%)
2023-01-11T22:03:20.2701439Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215655.xml (deflated 40%)
2023-01-11T22:03:20.2702228Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215659.xml (deflated 39%)
2023-01-11T22:03:20.2702814Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215704.xml (deflated 40%)
2023-01-11T22:03:20.2703406Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20230111215708.xml (deflated 40%)
2023-01-11T22:03:20.2704087Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215713.xml (deflated 44%)
2023-01-11T22:03:20.2704828Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215717.xml (deflated 44%)
2023-01-11T22:03:20.2705583Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215722.xml (deflated 44%)
2023-01-11T22:03:20.2706331Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215727.xml (deflated 44%)
2023-01-11T22:03:20.2707076Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215731.xml (deflated 44%)
2023-01-11T22:03:20.2707814Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215736.xml (deflated 44%)
2023-01-11T22:03:20.2708622Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215741.xml (deflated 43%)
2023-01-11T22:03:20.2709361Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215745.xml (deflated 43%)
2023-01-11T22:03:20.2710105Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215750.xml (deflated 44%)
2023-01-11T22:03:20.2710840Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215755.xml (deflated 43%)
2023-01-11T22:03:20.2711609Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215759.xml (deflated 43%)
2023-01-11T22:03:20.2712361Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215804.xml (deflated 44%)
2023-01-11T22:03:20.2713104Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215809.xml (deflated 44%)
2023-01-11T22:03:20.2713848Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215814.xml (deflated 44%)
2023-01-11T22:03:20.2714571Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215819.xml (deflated 44%)
2023-01-11T22:03:20.2715314Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215823.xml (deflated 44%)
2023-01-11T22:03:20.2716059Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215828.xml (deflated 44%)
2023-01-11T22:03:20.2716805Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215833.xml (deflated 44%)
2023-01-11T22:03:20.2717544Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215838.xml (deflated 43%)
2023-01-11T22:03:20.2718272Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215843.xml (deflated 43%)
2023-01-11T22:03:20.2719096Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215847.xml (deflated 43%)
2023-01-11T22:03:20.2719850Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215852.xml (deflated 44%)
2023-01-11T22:03:20.2720602Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215857.xml (deflated 44%)
2023-01-11T22:03:20.2721333Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215902.xml (deflated 44%)
2023-01-11T22:03:20.2722074Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215907.xml (deflated 44%)
2023-01-11T22:03:20.2722825Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215912.xml (deflated 43%)
2023-01-11T22:03:20.2723571Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215917.xml (deflated 44%)
2023-01-11T22:03:20.2724334Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215922.xml (deflated 44%)
2023-01-11T22:03:20.2725066Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215927.xml (deflated 44%)
2023-01-11T22:03:20.2725813Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215931.xml (deflated 44%)
2023-01-11T22:03:20.2726564Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20230111215936.xml (deflated 44%)
2023-01-11T22:03:20.2727309Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111215941.xml (deflated 43%)
2023-01-11T22:03:20.2727984Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111215945.xml (deflated 43%)
2023-01-11T22:03:20.2728677Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111215950.xml (deflated 43%)
2023-01-11T22:03:20.2729365Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111215955.xml (deflated 43%)
2023-01-11T22:03:20.2730050Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111215957.xml (deflated 43%)
2023-01-11T22:03:20.2730725Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220001.xml (deflated 44%)
2023-01-11T22:03:20.2731408Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220006.xml (deflated 44%)
2023-01-11T22:03:20.2732087Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220010.xml (deflated 43%)
2023-01-11T22:03:20.2732774Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220015.xml (deflated 43%)
2023-01-11T22:03:20.2733448Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220020.xml (deflated 43%)
2023-01-11T22:03:20.2734129Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220024.xml (deflated 43%)
2023-01-11T22:03:20.2734803Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220029.xml (deflated 43%)
2023-01-11T22:03:20.2735485Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220034.xml (deflated 43%)
2023-01-11T22:03:20.2736172Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220039.xml (deflated 43%)
2023-01-11T22:03:20.2736843Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220044.xml (deflated 43%)
2023-01-11T22:03:20.2737518Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220048.xml (deflated 43%)
2023-01-11T22:03:20.2738205Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220053.xml (deflated 43%)
2023-01-11T22:03:20.2739058Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220058.xml (deflated 43%)
2023-01-11T22:03:20.2739762Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220103.xml (deflated 43%)
2023-01-11T22:03:20.2740441Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220107.xml (deflated 43%)
2023-01-11T22:03:20.2741124Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220112.xml (deflated 43%)
2023-01-11T22:03:20.2741934Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220117.xml (deflated 43%)
2023-01-11T22:03:20.2742606Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220121.xml (deflated 44%)
2023-01-11T22:03:20.2743340Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220126.xml (deflated 43%)
2023-01-11T22:03:20.2744025Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220133.xml (deflated 43%)
2023-01-11T22:03:20.2744701Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220139.xml (deflated 43%)
2023-01-11T22:03:20.2745382Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220157.xml (deflated 43%)
2023-01-11T22:03:20.2746058Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220201.xml (deflated 44%)
2023-01-11T22:03:20.2746739Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220206.xml (deflated 43%)
2023-01-11T22:03:20.2747426Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220211.xml (deflated 44%)
2023-01-11T22:03:20.2748107Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220215.xml (deflated 44%)
2023-01-11T22:03:20.2748774Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220220.xml (deflated 44%)
2023-01-11T22:03:20.2749454Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220225.xml (deflated 43%)
2023-01-11T22:03:20.2750128Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220230.xml (deflated 44%)
2023-01-11T22:03:20.2750806Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220234.xml (deflated 45%)
2023-01-11T22:03:20.2751470Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220239.xml (deflated 44%)
2023-01-11T22:03:20.2752145Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220243.xml (deflated 43%)
2023-01-11T22:03:20.2752813Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220248.xml (deflated 43%)
2023-01-11T22:03:20.2753496Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20230111220253.xml (deflated 43%)
2023-01-11T22:03:20.2754200Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20230111220257.xml (deflated 44%)
2023-01-11T22:03:20.2754915Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20230111220302.xml (deflated 43%)
2023-01-11T22:03:20.2755678Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20230111220306.xml (deflated 43%)
2023-01-11T22:03:20.2756378Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204719.xml (deflated 41%)
2023-01-11T22:03:20.2757051Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204723.xml (deflated 41%)
2023-01-11T22:03:20.2757713Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204726.xml (deflated 42%)
2023-01-11T22:03:20.2758415Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204728.xml (deflated 42%)
2023-01-11T22:03:20.2759157Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204729.xml (deflated 42%)
2023-01-11T22:03:20.2759833Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204731.xml (deflated 42%)
2023-01-11T22:03:20.2760499Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204734.xml (deflated 41%)
2023-01-11T22:03:20.2761169Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204737.xml (deflated 41%)
2023-01-11T22:03:20.2761836Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204741.xml (deflated 42%)
2023-01-11T22:03:20.2762504Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204742.xml (deflated 42%)
2023-01-11T22:03:20.2763173Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204744.xml (deflated 42%)
2023-01-11T22:03:20.2763821Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111204745.xml (deflated 42%)
2023-01-11T22:03:20.2764456Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204752.xml (deflated 42%)
2023-01-11T22:03:20.2765055Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204756.xml (deflated 42%)
2023-01-11T22:03:20.2765658Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204758.xml (deflated 42%)
2023-01-11T22:03:20.2766246Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204802.xml (deflated 42%)
2023-01-11T22:03:20.2766840Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204804.xml (deflated 42%)
2023-01-11T22:03:20.2767430Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204808.xml (deflated 41%)
2023-01-11T22:03:20.2768022Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204812.xml (deflated 42%)
2023-01-11T22:03:20.2768609Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204816.xml (deflated 41%)
2023-01-11T22:03:20.2769200Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204820.xml (deflated 42%)
2023-01-11T22:03:20.2769793Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204824.xml (deflated 41%)
2023-01-11T22:03:20.2770427Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204828.xml (deflated 41%)
2023-01-11T22:03:20.2771006Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204832.xml (deflated 42%)
2023-01-11T22:03:20.2771588Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204836.xml (deflated 41%)
2023-01-11T22:03:20.2772178Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204840.xml (deflated 42%)
2023-01-11T22:03:20.2772769Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204842.xml (deflated 42%)
2023-01-11T22:03:20.2773377Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204846.xml (deflated 42%)
2023-01-11T22:03:20.2773970Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204848.xml (deflated 43%)
2023-01-11T22:03:20.2774553Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204852.xml (deflated 42%)
2023-01-11T22:03:20.2775142Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204854.xml (deflated 46%)
2023-01-11T22:03:20.2775717Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204858.xml (deflated 47%)
2023-01-11T22:03:20.2776303Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204902.xml (deflated 48%)
2023-01-11T22:03:20.2776892Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204906.xml (deflated 46%)
2023-01-11T22:03:20.2777481Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204910.xml (deflated 41%)
2023-01-11T22:03:20.2778055Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204914.xml (deflated 40%)
2023-01-11T22:03:20.2778641Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204919.xml (deflated 41%)
2023-01-11T22:03:20.2779227Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204923.xml (deflated 41%)
2023-01-11T22:03:20.2779818Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204927.xml (deflated 41%)
2023-01-11T22:03:20.2780394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204931.xml (deflated 41%)
2023-01-11T22:03:20.2780984Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204935.xml (deflated 40%)
2023-01-11T22:03:20.2781572Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204939.xml (deflated 43%)
2023-01-11T22:03:20.2782311Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204941.xml (deflated 42%)
2023-01-11T22:03:20.2782892Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204943.xml (deflated 41%)
2023-01-11T22:03:20.2783485Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204947.xml (deflated 40%)
2023-01-11T22:03:20.2784075Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204951.xml (deflated 43%)
2023-01-11T22:03:20.2784665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204953.xml (deflated 43%)
2023-01-11T22:03:20.2785294Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204954.xml (deflated 43%)
2023-01-11T22:03:20.2785882Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204956.xml (deflated 42%)
2023-01-11T22:03:20.2786467Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111204958.xml (deflated 40%)
2023-01-11T22:03:20.2787054Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205002.xml (deflated 40%)
2023-01-11T22:03:20.2787632Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205006.xml (deflated 43%)
2023-01-11T22:03:20.2788256Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205008.xml (deflated 41%)
2023-01-11T22:03:20.2788847Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205012.xml (deflated 41%)
2023-01-11T22:03:20.2789431Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205016.xml (deflated 41%)
2023-01-11T22:03:20.2790007Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205020.xml (deflated 41%)
2023-01-11T22:03:20.2790595Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205024.xml (deflated 41%)
2023-01-11T22:03:20.2791184Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205028.xml (deflated 41%)
2023-01-11T22:03:20.2791769Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205032.xml (deflated 41%)
2023-01-11T22:03:20.2792345Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205037.xml (deflated 41%)
2023-01-11T22:03:20.2792933Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205041.xml (deflated 41%)
2023-01-11T22:03:20.2793518Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205045.xml (deflated 41%)
2023-01-11T22:03:20.2794110Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205049.xml (deflated 42%)
2023-01-11T22:03:20.2794697Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205053.xml (deflated 41%)
2023-01-11T22:03:20.2795274Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205057.xml (deflated 41%)
2023-01-11T22:03:20.2795866Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205101.xml (deflated 41%)
2023-01-11T22:03:20.2796453Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205105.xml (deflated 41%)
2023-01-11T22:03:20.2797042Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205109.xml (deflated 41%)
2023-01-11T22:03:20.2797617Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205114.xml (deflated 41%)
2023-01-11T22:03:20.2798206Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205118.xml (deflated 41%)
2023-01-11T22:03:20.2798797Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205122.xml (deflated 41%)
2023-01-11T22:03:20.2799515Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205126.xml (deflated 41%)
2023-01-11T22:03:20.2800097Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205130.xml (deflated 41%)
2023-01-11T22:03:20.2800689Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205134.xml (deflated 41%)
2023-01-11T22:03:20.2801274Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205138.xml (deflated 41%)
2023-01-11T22:03:20.2801861Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205142.xml (deflated 41%)
2023-01-11T22:03:20.2802473Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205146.xml (deflated 42%)
2023-01-11T22:03:20.2803065Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205150.xml (deflated 42%)
2023-01-11T22:03:20.2803650Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205154.xml (deflated 41%)
2023-01-11T22:03:20.2804234Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205159.xml (deflated 42%)
2023-01-11T22:03:20.2804811Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205203.xml (deflated 41%)
2023-01-11T22:03:20.2805395Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205207.xml (deflated 41%)
2023-01-11T22:03:20.2805982Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205211.xml (deflated 41%)
2023-01-11T22:03:20.2806567Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205215.xml (deflated 42%)
2023-01-11T22:03:20.2807138Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205219.xml (deflated 42%)
2023-01-11T22:03:20.2807721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205223.xml (deflated 41%)
2023-01-11T22:03:20.2808306Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205227.xml (deflated 43%)
2023-01-11T22:03:20.2808888Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205229.xml (deflated 43%)
2023-01-11T22:03:20.2809461Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205231.xml (deflated 43%)
2023-01-11T22:03:20.2810046Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205232.xml (deflated 42%)
2023-01-11T22:03:20.2810635Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205234.xml (deflated 43%)
2023-01-11T22:03:20.2811214Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205236.xml (deflated 42%)
2023-01-11T22:03:20.2811784Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205237.xml (deflated 43%)
2023-01-11T22:03:20.2812363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205239.xml (deflated 42%)
2023-01-11T22:03:20.2812946Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205241.xml (deflated 43%)
2023-01-11T22:03:20.2813535Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205242.xml (deflated 43%)
2023-01-11T22:03:20.2814139Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205244.xml (deflated 43%)
2023-01-11T22:03:20.2814722Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205246.xml (deflated 43%)
2023-01-11T22:03:20.2815304Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205247.xml (deflated 43%)
2023-01-11T22:03:20.2815894Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205249.xml (deflated 43%)
2023-01-11T22:03:20.2816468Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205251.xml (deflated 43%)
2023-01-11T22:03:20.2817084Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205253.xml (deflated 43%)
2023-01-11T22:03:20.2817678Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205254.xml (deflated 43%)
2023-01-11T22:03:20.2818265Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205256.xml (deflated 43%)
2023-01-11T22:03:20.2818841Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205258.xml (deflated 43%)
2023-01-11T22:03:20.2819433Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205259.xml (deflated 43%)
2023-01-11T22:03:20.2820016Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205301.xml (deflated 43%)
2023-01-11T22:03:20.2820605Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205303.xml (deflated 43%)
2023-01-11T22:03:20.2821179Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205304.xml (deflated 43%)
2023-01-11T22:03:20.2821860Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205306.xml (deflated 43%)
2023-01-11T22:03:20.2822451Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205308.xml (deflated 42%)
2023-01-11T22:03:20.2823052Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205312.xml (deflated 42%)
2023-01-11T22:03:20.2823730Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205313.xml (deflated 42%)
2023-01-11T22:03:20.2824320Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205315.xml (deflated 41%)
2023-01-11T22:03:20.2824911Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205320.xml (deflated 42%)
2023-01-11T22:03:20.2825501Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205325.xml (deflated 41%)
2023-01-11T22:03:20.2826076Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205330.xml (deflated 42%)
2023-01-11T22:03:20.2826664Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205334.xml (deflated 41%)
2023-01-11T22:03:20.2827252Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205339.xml (deflated 42%)
2023-01-11T22:03:20.2827848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205343.xml (deflated 40%)
2023-01-11T22:03:20.2828424Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205348.xml (deflated 42%)
2023-01-11T22:03:20.2829070Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205350.xml (deflated 41%)
2023-01-11T22:03:20.2829654Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205359.xml (deflated 41%)
2023-01-11T22:03:20.2830245Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205403.xml (deflated 40%)
2023-01-11T22:03:20.2830829Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205407.xml (deflated 42%)
2023-01-11T22:03:20.2831404Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205408.xml (deflated 42%)
2023-01-11T22:03:20.2832030Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205410.xml (deflated 41%)
2023-01-11T22:03:20.2832622Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205412.xml (deflated 41%)
2023-01-11T22:03:20.2833207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205414.xml (deflated 41%)
2023-01-11T22:03:20.2833778Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205415.xml (deflated 41%)
2023-01-11T22:03:20.2834364Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205417.xml (deflated 42%)
2023-01-11T22:03:20.2834949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205419.xml (deflated 41%)
2023-01-11T22:03:20.2835540Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205420.xml (deflated 41%)
2023-01-11T22:03:20.2836120Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205424.xml (deflated 42%)
2023-01-11T22:03:20.2836703Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205426.xml (deflated 40%)
2023-01-11T22:03:20.2837290Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205430.xml (deflated 41%)
2023-01-11T22:03:20.2837873Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205435.xml (deflated 42%)
2023-01-11T22:03:20.2838535Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205439.xml (deflated 42%)
2023-01-11T22:03:20.2839210Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205441.xml (deflated 41%)
2023-01-11T22:03:20.2839806Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205442.xml (deflated 41%)
2023-01-11T22:03:20.2840392Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205444.xml (deflated 42%)
2023-01-11T22:03:20.2840973Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205448.xml (deflated 41%)
2023-01-11T22:03:20.2841558Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205452.xml (deflated 42%)
2023-01-11T22:03:20.2842144Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205456.xml (deflated 42%)
2023-01-11T22:03:20.2842736Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205500.xml (deflated 42%)
2023-01-11T22:03:20.2843349Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205504.xml (deflated 42%)
2023-01-11T22:03:20.2843935Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205506.xml (deflated 42%)
2023-01-11T22:03:20.2844519Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205508.xml (deflated 41%)
2023-01-11T22:03:20.2845106Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205510.xml (deflated 41%)
2023-01-11T22:03:20.2845675Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205514.xml (deflated 41%)
2023-01-11T22:03:20.2846291Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205518.xml (deflated 41%)
2023-01-11T22:03:20.2846883Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205520.xml (deflated 41%)
2023-01-11T22:03:20.2847463Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205522.xml (deflated 41%)
2023-01-11T22:03:20.2848033Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205526.xml (deflated 41%)
2023-01-11T22:03:20.2848619Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205527.xml (deflated 41%)
2023-01-11T22:03:20.2849202Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205529.xml (deflated 42%)
2023-01-11T22:03:20.2849785Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205531.xml (deflated 42%)
2023-01-11T22:03:20.2850362Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205533.xml (deflated 41%)
2023-01-11T22:03:20.2850945Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205537.xml (deflated 41%)
2023-01-11T22:03:20.2851533Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205541.xml (deflated 42%)
2023-01-11T22:03:20.2852119Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205543.xml (deflated 41%)
2023-01-11T22:03:20.2852689Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205547.xml (deflated 42%)
2023-01-11T22:03:20.2853271Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205551.xml (deflated 42%)
2023-01-11T22:03:20.2853858Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205555.xml (deflated 42%)
2023-01-11T22:03:20.2854444Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205559.xml (deflated 42%)
2023-01-11T22:03:20.2855012Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205603.xml (deflated 41%)
2023-01-11T22:03:20.2855599Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205607.xml (deflated 41%)
2023-01-11T22:03:20.2856184Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205611.xml (deflated 42%)
2023-01-11T22:03:20.2856768Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205615.xml (deflated 41%)
2023-01-11T22:03:20.2857342Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205619.xml (deflated 42%)
2023-01-11T22:03:20.2857962Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205623.xml (deflated 42%)
2023-01-11T22:03:20.2858543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205627.xml (deflated 42%)
2023-01-11T22:03:20.2859130Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205631.xml (deflated 42%)
2023-01-11T22:03:20.2859703Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205635.xml (deflated 41%)
2023-01-11T22:03:20.2860288Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205637.xml (deflated 42%)
2023-01-11T22:03:20.2860899Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205641.xml (deflated 41%)
2023-01-11T22:03:20.2861488Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205645.xml (deflated 40%)
2023-01-11T22:03:20.2862207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205649.xml (deflated 42%)
2023-01-11T22:03:20.2862798Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205653.xml (deflated 41%)
2023-01-11T22:03:20.2863380Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205655.xml (deflated 41%)
2023-01-11T22:03:20.2863967Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205657.xml (deflated 42%)
2023-01-11T22:03:20.2864540Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205659.xml (deflated 42%)
2023-01-11T22:03:20.2865133Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205700.xml (deflated 42%)
2023-01-11T22:03:20.2865721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205702.xml (deflated 42%)
2023-01-11T22:03:20.2866304Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205706.xml (deflated 42%)
2023-01-11T22:03:20.2866891Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205708.xml (deflated 42%)
2023-01-11T22:03:20.2867463Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205710.xml (deflated 42%)
2023-01-11T22:03:20.2868053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205711.xml (deflated 42%)
2023-01-11T22:03:20.2868642Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205715.xml (deflated 42%)
2023-01-11T22:03:20.2869225Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205717.xml (deflated 41%)
2023-01-11T22:03:20.2869796Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205719.xml (deflated 42%)
2023-01-11T22:03:20.2870381Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205721.xml (deflated 42%)
2023-01-11T22:03:20.2870967Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205725.xml (deflated 42%)
2023-01-11T22:03:20.2871549Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205729.xml (deflated 42%)
2023-01-11T22:03:20.2872123Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205733.xml (deflated 42%)
2023-01-11T22:03:20.2872769Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205735.xml (deflated 42%)
2023-01-11T22:03:20.2873356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205737.xml (deflated 42%)
2023-01-11T22:03:20.2873941Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205739.xml (deflated 41%)
2023-01-11T22:03:20.2874514Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205741.xml (deflated 42%)
2023-01-11T22:03:20.2875098Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205745.xml (deflated 41%)
2023-01-11T22:03:20.2875727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205749.xml (deflated 41%)
2023-01-11T22:03:20.2876320Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205754.xml (deflated 42%)
2023-01-11T22:03:20.2876894Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205756.xml (deflated 41%)
2023-01-11T22:03:20.2877477Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205757.xml (deflated 40%)
2023-01-11T22:03:20.2878063Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205801.xml (deflated 41%)
2023-01-11T22:03:20.2878646Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205806.xml (deflated 41%)
2023-01-11T22:03:20.2879285Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205810.xml (deflated 43%)
2023-01-11T22:03:20.2879878Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205811.xml (deflated 41%)
2023-01-11T22:03:20.2880463Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205815.xml (deflated 41%)
2023-01-11T22:03:20.2881053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205819.xml (deflated 40%)
2023-01-11T22:03:20.2881625Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205824.xml (deflated 42%)
2023-01-11T22:03:20.2882213Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205826.xml (deflated 41%)
2023-01-11T22:03:20.2882800Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205830.xml (deflated 41%)
2023-01-11T22:03:20.2883386Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205834.xml (deflated 41%)
2023-01-11T22:03:20.2883959Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205838.xml (deflated 40%)
2023-01-11T22:03:20.2884543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205842.xml (deflated 40%)
2023-01-11T22:03:20.2885128Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205846.xml (deflated 41%)
2023-01-11T22:03:20.2885711Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205848.xml (deflated 41%)
2023-01-11T22:03:20.2886285Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205852.xml (deflated 41%)
2023-01-11T22:03:20.2886902Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205856.xml (deflated 41%)
2023-01-11T22:03:20.2887487Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205900.xml (deflated 41%)
2023-01-11T22:03:20.2888072Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205904.xml (deflated 41%)
2023-01-11T22:03:20.2888646Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205906.xml (deflated 41%)
2023-01-11T22:03:20.2889228Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205908.xml (deflated 42%)
2023-01-11T22:03:20.2889842Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205914.xml (deflated 41%)
2023-01-11T22:03:20.2890435Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205920.xml (deflated 41%)
2023-01-11T22:03:20.2891006Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205924.xml (deflated 40%)
2023-01-11T22:03:20.2891592Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205928.xml (deflated 40%)
2023-01-11T22:03:20.2892173Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205933.xml (deflated 42%)
2023-01-11T22:03:20.2892755Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205934.xml (deflated 42%)
2023-01-11T22:03:20.2893329Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205936.xml (deflated 42%)
2023-01-11T22:03:20.2893920Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205938.xml (deflated 42%)
2023-01-11T22:03:20.2894508Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205939.xml (deflated 43%)
2023-01-11T22:03:20.2895095Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205941.xml (deflated 42%)
2023-01-11T22:03:20.2895661Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205943.xml (deflated 42%)
2023-01-11T22:03:20.2896247Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205944.xml (deflated 42%)
2023-01-11T22:03:20.2896835Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205946.xml (deflated 41%)
2023-01-11T22:03:20.2897421Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205950.xml (deflated 42%)
2023-01-11T22:03:20.2897993Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205954.xml (deflated 42%)
2023-01-11T22:03:20.2898577Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205956.xml (deflated 43%)
2023-01-11T22:03:20.2899158Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111205958.xml (deflated 42%)
2023-01-11T22:03:20.2899742Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210000.xml (deflated 42%)
2023-01-11T22:03:20.2900310Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210004.xml (deflated 42%)
2023-01-11T22:03:20.2900896Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210008.xml (deflated 42%)
2023-01-11T22:03:20.2901528Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210012.xml (deflated 42%)
2023-01-11T22:03:20.2902204Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210014.xml (deflated 42%)
2023-01-11T22:03:20.2902794Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210016.xml (deflated 41%)
2023-01-11T22:03:20.2903363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210020.xml (deflated 41%)
2023-01-11T22:03:20.2903949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210024.xml (deflated 42%)
2023-01-11T22:03:20.2904587Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210026.xml (deflated 41%)
2023-01-11T22:03:20.2905176Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210030.xml (deflated 41%)
2023-01-11T22:03:20.2905749Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210034.xml (deflated 41%)
2023-01-11T22:03:20.2906334Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210038.xml (deflated 41%)
2023-01-11T22:03:20.2906923Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210042.xml (deflated 41%)
2023-01-11T22:03:20.2907505Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210047.xml (deflated 41%)
2023-01-11T22:03:20.2908078Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210051.xml (deflated 41%)
2023-01-11T22:03:20.2908666Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210055.xml (deflated 41%)
2023-01-11T22:03:20.2909249Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210059.xml (deflated 41%)
2023-01-11T22:03:20.2909833Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210103.xml (deflated 41%)
2023-01-11T22:03:20.2910405Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210107.xml (deflated 43%)
2023-01-11T22:03:20.2910985Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210109.xml (deflated 42%)
2023-01-11T22:03:20.2911571Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210113.xml (deflated 43%)
2023-01-11T22:03:20.2912157Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210115.xml (deflated 44%)
2023-01-11T22:03:20.2912730Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210116.xml (deflated 41%)
2023-01-11T22:03:20.2913315Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210121.xml (deflated 43%)
2023-01-11T22:03:20.2913897Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210122.xml (deflated 43%)
2023-01-11T22:03:20.2914482Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210124.xml (deflated 41%)
2023-01-11T22:03:20.2915055Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210128.xml (deflated 42%)
2023-01-11T22:03:20.2915644Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210132.xml (deflated 41%)
2023-01-11T22:03:20.2916272Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210136.xml (deflated 41%)
2023-01-11T22:03:20.2916856Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210140.xml (deflated 42%)
2023-01-11T22:03:20.2917617Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210142.xml (deflated 42%)
2023-01-11T22:03:20.2918212Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210144.xml (deflated 41%)
2023-01-11T22:03:20.2918793Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210148.xml (deflated 41%)
2023-01-11T22:03:20.2919483Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210152.xml (deflated 41%)
2023-01-11T22:03:20.2920067Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210156.xml (deflated 41%)
2023-01-11T22:03:20.2920653Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210200.xml (deflated 41%)
2023-01-11T22:03:20.2921238Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210204.xml (deflated 41%)
2023-01-11T22:03:20.2921826Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210208.xml (deflated 41%)
2023-01-11T22:03:20.2922399Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210212.xml (deflated 41%)
2023-01-11T22:03:20.2922987Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210216.xml (deflated 42%)
2023-01-11T22:03:20.2923574Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210218.xml (deflated 42%)
2023-01-11T22:03:20.2924160Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210220.xml (deflated 42%)
2023-01-11T22:03:20.2924729Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210221.xml (deflated 41%)
2023-01-11T22:03:20.2925314Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210225.xml (deflated 41%)
2023-01-11T22:03:20.2925901Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210229.xml (deflated 41%)
2023-01-11T22:03:20.2926491Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210233.xml (deflated 41%)
2023-01-11T22:03:20.2927066Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210237.xml (deflated 41%)
2023-01-11T22:03:20.2927653Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210241.xml (deflated 42%)
2023-01-11T22:03:20.2928237Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210246.xml (deflated 42%)
2023-01-11T22:03:20.2928820Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210250.xml (deflated 41%)
2023-01-11T22:03:20.2929392Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210254.xml (deflated 42%)
2023-01-11T22:03:20.2929984Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210258.xml (deflated 43%)
2023-01-11T22:03:20.2930601Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210302.xml (deflated 41%)
2023-01-11T22:03:20.2931188Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210304.xml (deflated 41%)
2023-01-11T22:03:20.2931757Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210307.xml (deflated 43%)
2023-01-11T22:03:20.2932345Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210311.xml (deflated 42%)
2023-01-11T22:03:20.2932933Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210313.xml (deflated 43%)
2023-01-11T22:03:20.2933548Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210317.xml (deflated 43%)
2023-01-11T22:03:20.2934122Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210319.xml (deflated 42%)
2023-01-11T22:03:20.2934706Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210323.xml (deflated 41%)
2023-01-11T22:03:20.2935291Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210327.xml (deflated 42%)
2023-01-11T22:03:20.2935875Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210331.xml (deflated 41%)
2023-01-11T22:03:20.2936444Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210335.xml (deflated 41%)
2023-01-11T22:03:20.2937027Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210339.xml (deflated 41%)
2023-01-11T22:03:20.2937615Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210343.xml (deflated 41%)
2023-01-11T22:03:20.2938201Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210347.xml (deflated 41%)
2023-01-11T22:03:20.2938788Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210351.xml (deflated 41%)
2023-01-11T22:03:20.2939360Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210355.xml (deflated 42%)
2023-01-11T22:03:20.2939945Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210358.xml (deflated 42%)
2023-01-11T22:03:20.2940531Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210402.xml (deflated 42%)
2023-01-11T22:03:20.2941118Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210404.xml (deflated 43%)
2023-01-11T22:03:20.2941808Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210408.xml (deflated 42%)
2023-01-11T22:03:20.2942438Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210409.xml (deflated 46%)
2023-01-11T22:03:20.2943025Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210413.xml (deflated 47%)
2023-01-11T22:03:20.2943614Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210417.xml (deflated 48%)
2023-01-11T22:03:20.2944188Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210421.xml (deflated 46%)
2023-01-11T22:03:20.2944776Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210425.xml (deflated 41%)
2023-01-11T22:03:20.2945415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210429.xml (deflated 41%)
2023-01-11T22:03:20.2945997Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210434.xml (deflated 41%)
2023-01-11T22:03:20.2946570Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210438.xml (deflated 41%)
2023-01-11T22:03:20.2947157Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210442.xml (deflated 41%)
2023-01-11T22:03:20.2947741Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210446.xml (deflated 41%)
2023-01-11T22:03:20.2948363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210450.xml (deflated 41%)
2023-01-11T22:03:20.2948942Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210454.xml (deflated 43%)
2023-01-11T22:03:20.2949523Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210456.xml (deflated 42%)
2023-01-11T22:03:20.2950108Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210457.xml (deflated 41%)
2023-01-11T22:03:20.2950691Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210502.xml (deflated 41%)
2023-01-11T22:03:20.2951264Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210506.xml (deflated 43%)
2023-01-11T22:03:20.2951849Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210507.xml (deflated 43%)
2023-01-11T22:03:20.2952439Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210509.xml (deflated 43%)
2023-01-11T22:03:20.2953022Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210511.xml (deflated 43%)
2023-01-11T22:03:20.2953592Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210512.xml (deflated 40%)
2023-01-11T22:03:20.2954176Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210516.xml (deflated 40%)
2023-01-11T22:03:20.2954758Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210521.xml (deflated 44%)
2023-01-11T22:03:20.2955342Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210522.xml (deflated 41%)
2023-01-11T22:03:20.2955912Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210527.xml (deflated 41%)
2023-01-11T22:03:20.2956496Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210531.xml (deflated 41%)
2023-01-11T22:03:20.2957078Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210535.xml (deflated 41%)
2023-01-11T22:03:20.2957664Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210539.xml (deflated 41%)
2023-01-11T22:03:20.2958231Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210543.xml (deflated 40%)
2023-01-11T22:03:20.2958867Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210547.xml (deflated 41%)
2023-01-11T22:03:20.2959466Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210551.xml (deflated 41%)
2023-01-11T22:03:20.2960090Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210555.xml (deflated 41%)
2023-01-11T22:03:20.2960665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210600.xml (deflated 41%)
2023-01-11T22:03:20.2961252Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210604.xml (deflated 41%)
2023-01-11T22:03:20.2961837Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210608.xml (deflated 41%)
2023-01-11T22:03:20.2962419Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210612.xml (deflated 41%)
2023-01-11T22:03:20.2963027Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210616.xml (deflated 41%)
2023-01-11T22:03:20.2963622Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210620.xml (deflated 41%)
2023-01-11T22:03:20.2964205Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210624.xml (deflated 41%)
2023-01-11T22:03:20.2964787Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210628.xml (deflated 41%)
2023-01-11T22:03:20.2965359Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210633.xml (deflated 41%)
2023-01-11T22:03:20.2965941Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210637.xml (deflated 40%)
2023-01-11T22:03:20.2966526Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210641.xml (deflated 41%)
2023-01-11T22:03:20.2967118Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210645.xml (deflated 41%)
2023-01-11T22:03:20.2967696Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210649.xml (deflated 41%)
2023-01-11T22:03:20.2968282Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210653.xml (deflated 41%)
2023-01-11T22:03:20.2968869Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210657.xml (deflated 41%)
2023-01-11T22:03:20.2969455Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210701.xml (deflated 42%)
2023-01-11T22:03:20.2970027Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210705.xml (deflated 41%)
2023-01-11T22:03:20.2970617Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210709.xml (deflated 41%)
2023-01-11T22:03:20.2971200Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210714.xml (deflated 42%)
2023-01-11T22:03:20.2971779Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210718.xml (deflated 41%)
2023-01-11T22:03:20.2972364Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210722.xml (deflated 41%)
2023-01-11T22:03:20.2972939Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210726.xml (deflated 41%)
2023-01-11T22:03:20.2973522Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210730.xml (deflated 42%)
2023-01-11T22:03:20.2974145Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210734.xml (deflated 42%)
2023-01-11T22:03:20.2974721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210738.xml (deflated 41%)
2023-01-11T22:03:20.2975296Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210742.xml (deflated 43%)
2023-01-11T22:03:20.2975876Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210744.xml (deflated 43%)
2023-01-11T22:03:20.2976561Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210745.xml (deflated 43%)
2023-01-11T22:03:20.2977179Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210747.xml (deflated 42%)
2023-01-11T22:03:20.2977769Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210749.xml (deflated 43%)
2023-01-11T22:03:20.2978357Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210750.xml (deflated 42%)
2023-01-11T22:03:20.2978928Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210752.xml (deflated 43%)
2023-01-11T22:03:20.2979513Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210754.xml (deflated 42%)
2023-01-11T22:03:20.2980098Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210755.xml (deflated 43%)
2023-01-11T22:03:20.2980686Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210757.xml (deflated 43%)
2023-01-11T22:03:20.2981263Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210759.xml (deflated 43%)
2023-01-11T22:03:20.2981952Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210800.xml (deflated 43%)
2023-01-11T22:03:20.2982536Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210802.xml (deflated 43%)
2023-01-11T22:03:20.2983122Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210804.xml (deflated 43%)
2023-01-11T22:03:20.2983690Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210805.xml (deflated 43%)
2023-01-11T22:03:20.2984274Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210807.xml (deflated 43%)
2023-01-11T22:03:20.2984858Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210809.xml (deflated 43%)
2023-01-11T22:03:20.2985445Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210810.xml (deflated 43%)
2023-01-11T22:03:20.2986021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210812.xml (deflated 43%)
2023-01-11T22:03:20.2986605Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210814.xml (deflated 43%)
2023-01-11T22:03:20.2987181Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210816.xml (deflated 43%)
2023-01-11T22:03:20.2987765Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210817.xml (deflated 43%)
2023-01-11T22:03:20.2988338Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210819.xml (deflated 43%)
2023-01-11T22:03:20.2988984Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210821.xml (deflated 43%)
2023-01-11T22:03:20.2989569Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210822.xml (deflated 42%)
2023-01-11T22:03:20.2990154Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210826.xml (deflated 42%)
2023-01-11T22:03:20.2990720Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210828.xml (deflated 42%)
2023-01-11T22:03:20.2991311Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210830.xml (deflated 41%)
2023-01-11T22:03:20.2991934Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210835.xml (deflated 42%)
2023-01-11T22:03:20.2992525Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210839.xml (deflated 41%)
2023-01-11T22:03:20.2993093Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210844.xml (deflated 41%)
2023-01-11T22:03:20.2993683Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210848.xml (deflated 41%)
2023-01-11T22:03:20.2994268Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210853.xml (deflated 42%)
2023-01-11T22:03:20.2994856Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210857.xml (deflated 41%)
2023-01-11T22:03:20.2995431Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210902.xml (deflated 41%)
2023-01-11T22:03:20.2996023Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210908.xml (deflated 41%)
2023-01-11T22:03:20.2996603Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210917.xml (deflated 41%)
2023-01-11T22:03:20.2997188Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210921.xml (deflated 41%)
2023-01-11T22:03:20.2997757Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210925.xml (deflated 42%)
2023-01-11T22:03:20.2998344Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210926.xml (deflated 42%)
2023-01-11T22:03:20.2998993Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210928.xml (deflated 41%)
2023-01-11T22:03:20.2999584Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210930.xml (deflated 42%)
2023-01-11T22:03:20.3000158Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210932.xml (deflated 42%)
2023-01-11T22:03:20.3000746Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210933.xml (deflated 41%)
2023-01-11T22:03:20.3001329Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210935.xml (deflated 42%)
2023-01-11T22:03:20.3001924Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210937.xml (deflated 42%)
2023-01-11T22:03:20.3002510Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210938.xml (deflated 41%)
2023-01-11T22:03:20.3003086Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210942.xml (deflated 42%)
2023-01-11T22:03:20.3003731Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210944.xml (deflated 40%)
2023-01-11T22:03:20.3004319Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210949.xml (deflated 41%)
2023-01-11T22:03:20.3004909Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210953.xml (deflated 42%)
2023-01-11T22:03:20.3005483Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210957.xml (deflated 42%)
2023-01-11T22:03:20.3006074Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111210959.xml (deflated 41%)
2023-01-11T22:03:20.3006694Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211000.xml (deflated 42%)
2023-01-11T22:03:20.3007286Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211002.xml (deflated 42%)
2023-01-11T22:03:20.3007864Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211006.xml (deflated 42%)
2023-01-11T22:03:20.3008449Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211010.xml (deflated 42%)
2023-01-11T22:03:20.3009031Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211014.xml (deflated 42%)
2023-01-11T22:03:20.3009619Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211018.xml (deflated 42%)
2023-01-11T22:03:20.3010193Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211022.xml (deflated 42%)
2023-01-11T22:03:20.3010778Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211024.xml (deflated 42%)
2023-01-11T22:03:20.3011365Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211026.xml (deflated 42%)
2023-01-11T22:03:20.3011951Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211028.xml (deflated 41%)
2023-01-11T22:03:20.3012527Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211032.xml (deflated 42%)
2023-01-11T22:03:20.3013117Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211036.xml (deflated 41%)
2023-01-11T22:03:20.3013703Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211038.xml (deflated 41%)
2023-01-11T22:03:20.3014294Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211040.xml (deflated 41%)
2023-01-11T22:03:20.3014866Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211044.xml (deflated 42%)
2023-01-11T22:03:20.3015450Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211045.xml (deflated 42%)
2023-01-11T22:03:20.3016034Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211047.xml (deflated 42%)
2023-01-11T22:03:20.3016620Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211049.xml (deflated 42%)
2023-01-11T22:03:20.3017197Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211051.xml (deflated 42%)
2023-01-11T22:03:20.3017840Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211055.xml (deflated 41%)
2023-01-11T22:03:20.3018422Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211059.xml (deflated 42%)
2023-01-11T22:03:20.3019010Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211101.xml (deflated 41%)
2023-01-11T22:03:20.3019577Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211105.xml (deflated 42%)
2023-01-11T22:03:20.3020157Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211109.xml (deflated 42%)
2023-01-11T22:03:20.3020778Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211113.xml (deflated 42%)
2023-01-11T22:03:20.3021369Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211117.xml (deflated 42%)
2023-01-11T22:03:20.3022115Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211121.xml (deflated 42%)
2023-01-11T22:03:20.3022708Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211125.xml (deflated 42%)
2023-01-11T22:03:20.3023297Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211129.xml (deflated 42%)
2023-01-11T22:03:20.3023889Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211133.xml (deflated 42%)
2023-01-11T22:03:20.3024461Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211137.xml (deflated 42%)
2023-01-11T22:03:20.3025051Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211141.xml (deflated 42%)
2023-01-11T22:03:20.3025640Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211145.xml (deflated 42%)
2023-01-11T22:03:20.3026220Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211149.xml (deflated 42%)
2023-01-11T22:03:20.3026796Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211153.xml (deflated 42%)
2023-01-11T22:03:20.3027380Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211155.xml (deflated 42%)
2023-01-11T22:03:20.3027965Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211159.xml (deflated 42%)
2023-01-11T22:03:20.3028551Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211203.xml (deflated 41%)
2023-01-11T22:03:20.3029130Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211207.xml (deflated 42%)
2023-01-11T22:03:20.3029718Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211211.xml (deflated 41%)
2023-01-11T22:03:20.3030306Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211213.xml (deflated 41%)
2023-01-11T22:03:20.3030898Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211215.xml (deflated 42%)
2023-01-11T22:03:20.3031481Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211216.xml (deflated 42%)
2023-01-11T22:03:20.3032075Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211218.xml (deflated 42%)
2023-01-11T22:03:20.3032751Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211220.xml (deflated 42%)
2023-01-11T22:03:20.3033341Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211224.xml (deflated 42%)
2023-01-11T22:03:20.3033914Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211226.xml (deflated 42%)
2023-01-11T22:03:20.3034507Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211227.xml (deflated 42%)
2023-01-11T22:03:20.3035095Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211229.xml (deflated 42%)
2023-01-11T22:03:20.3035726Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211233.xml (deflated 42%)
2023-01-11T22:03:20.3036310Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211235.xml (deflated 41%)
2023-01-11T22:03:20.3036897Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211237.xml (deflated 42%)
2023-01-11T22:03:20.3037478Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211239.xml (deflated 42%)
2023-01-11T22:03:20.3038063Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211243.xml (deflated 42%)
2023-01-11T22:03:20.3038643Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211247.xml (deflated 42%)
2023-01-11T22:03:20.3039292Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211251.xml (deflated 42%)
2023-01-11T22:03:20.3039884Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211253.xml (deflated 42%)
2023-01-11T22:03:20.3040475Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211255.xml (deflated 42%)
2023-01-11T22:03:20.3041062Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211257.xml (deflated 41%)
2023-01-11T22:03:20.3041635Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211259.xml (deflated 42%)
2023-01-11T22:03:20.3042221Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211303.xml (deflated 41%)
2023-01-11T22:03:20.3042810Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211307.xml (deflated 41%)
2023-01-11T22:03:20.3043389Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211311.xml (deflated 42%)
2023-01-11T22:03:20.3043967Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211313.xml (deflated 42%)
2023-01-11T22:03:20.3044554Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211315.xml (deflated 41%)
2023-01-11T22:03:20.3045138Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211319.xml (deflated 41%)
2023-01-11T22:03:20.3045723Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211323.xml (deflated 41%)
2023-01-11T22:03:20.3046291Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211327.xml (deflated 43%)
2023-01-11T22:03:20.3046875Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211329.xml (deflated 41%)
2023-01-11T22:03:20.3047497Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211333.xml (deflated 41%)
2023-01-11T22:03:20.3048081Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211337.xml (deflated 41%)
2023-01-11T22:03:20.3048652Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211341.xml (deflated 42%)
2023-01-11T22:03:20.3049233Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211343.xml (deflated 41%)
2023-01-11T22:03:20.3049815Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211347.xml (deflated 41%)
2023-01-11T22:03:20.3050429Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211352.xml (deflated 41%)
2023-01-11T22:03:20.3051005Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211356.xml (deflated 41%)
2023-01-11T22:03:20.3051589Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211400.xml (deflated 41%)
2023-01-11T22:03:20.3052177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211404.xml (deflated 41%)
2023-01-11T22:03:20.3052757Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211405.xml (deflated 41%)
2023-01-11T22:03:20.3053326Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211410.xml (deflated 42%)
2023-01-11T22:03:20.3053914Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211414.xml (deflated 41%)
2023-01-11T22:03:20.3054501Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211418.xml (deflated 41%)
2023-01-11T22:03:20.3055087Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211422.xml (deflated 42%)
2023-01-11T22:03:20.3055656Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211424.xml (deflated 41%)
2023-01-11T22:03:20.3056238Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211425.xml (deflated 42%)
2023-01-11T22:03:20.3056823Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211431.xml (deflated 41%)
2023-01-11T22:03:20.3057410Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211438.xml (deflated 40%)
2023-01-11T22:03:20.3057990Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211442.xml (deflated 41%)
2023-01-11T22:03:20.3058571Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211446.xml (deflated 40%)
2023-01-11T22:03:20.3059152Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211450.xml (deflated 42%)
2023-01-11T22:03:20.3059735Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211452.xml (deflated 42%)
2023-01-11T22:03:20.3060306Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211454.xml (deflated 42%)
2023-01-11T22:03:20.3060890Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211455.xml (deflated 42%)
2023-01-11T22:03:20.3061524Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211457.xml (deflated 43%)
2023-01-11T22:03:20.3062219Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211459.xml (deflated 42%)
2023-01-11T22:03:20.3062793Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211500.xml (deflated 42%)
2023-01-11T22:03:20.3063375Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211502.xml (deflated 42%)
2023-01-11T22:03:20.3063964Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211504.xml (deflated 41%)
2023-01-11T22:03:20.3064598Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211508.xml (deflated 42%)
2023-01-11T22:03:20.3065171Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211512.xml (deflated 42%)
2023-01-11T22:03:20.3065761Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211513.xml (deflated 43%)
2023-01-11T22:03:20.3066346Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211515.xml (deflated 42%)
2023-01-11T22:03:20.3066928Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211517.xml (deflated 42%)
2023-01-11T22:03:20.3067503Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211521.xml (deflated 42%)
2023-01-11T22:03:20.3068088Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211525.xml (deflated 42%)
2023-01-11T22:03:20.3068673Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211529.xml (deflated 42%)
2023-01-11T22:03:20.3069266Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211531.xml (deflated 42%)
2023-01-11T22:03:20.3069838Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211533.xml (deflated 41%)
2023-01-11T22:03:20.3070427Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211537.xml (deflated 41%)
2023-01-11T22:03:20.3071016Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211541.xml (deflated 42%)
2023-01-11T22:03:20.3071602Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211543.xml (deflated 41%)
2023-01-11T22:03:20.3072187Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211547.xml (deflated 41%)
2023-01-11T22:03:20.3072759Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211552.xml (deflated 41%)
2023-01-11T22:03:20.3073340Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211556.xml (deflated 41%)
2023-01-11T22:03:20.3073924Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211600.xml (deflated 41%)
2023-01-11T22:03:20.3074508Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211604.xml (deflated 41%)
2023-01-11T22:03:20.3075083Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211608.xml (deflated 41%)
2023-01-11T22:03:20.3075669Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211612.xml (deflated 41%)
2023-01-11T22:03:20.3076311Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211617.xml (deflated 41%)
2023-01-11T22:03:20.3076893Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211621.xml (deflated 41%)
2023-01-11T22:03:20.3077469Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211625.xml (deflated 43%)
2023-01-11T22:03:20.3078053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211626.xml (deflated 41%)
2023-01-11T22:03:20.3078639Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211631.xml (deflated 43%)
2023-01-11T22:03:20.3079358Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211632.xml (deflated 44%)
2023-01-11T22:03:20.3079935Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211634.xml (deflated 41%)
2023-01-11T22:03:20.3080521Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211638.xml (deflated 43%)
2023-01-11T22:03:20.3081108Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211640.xml (deflated 43%)
2023-01-11T22:03:20.3081697Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211641.xml (deflated 42%)
2023-01-11T22:03:20.3082271Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211646.xml (deflated 41%)
2023-01-11T22:03:20.3082858Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211650.xml (deflated 41%)
2023-01-11T22:03:20.3083447Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211654.xml (deflated 41%)
2023-01-11T22:03:20.3084029Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211658.xml (deflated 42%)
2023-01-11T22:03:20.3084601Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211700.xml (deflated 42%)
2023-01-11T22:03:20.3085184Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211701.xml (deflated 41%)
2023-01-11T22:03:20.3085767Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211705.xml (deflated 41%)
2023-01-11T22:03:20.3086360Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211709.xml (deflated 41%)
2023-01-11T22:03:20.3086932Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211713.xml (deflated 41%)
2023-01-11T22:03:20.3087518Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211717.xml (deflated 41%)
2023-01-11T22:03:20.3088103Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211721.xml (deflated 41%)
2023-01-11T22:03:20.3088684Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211725.xml (deflated 41%)
2023-01-11T22:03:20.3089255Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211730.xml (deflated 41%)
2023-01-11T22:03:20.3089840Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211734.xml (deflated 42%)
2023-01-11T22:03:20.3090430Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211735.xml (deflated 42%)
2023-01-11T22:03:20.3091062Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211737.xml (deflated 41%)
2023-01-11T22:03:20.3091633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211739.xml (deflated 41%)
2023-01-11T22:03:20.3092211Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211743.xml (deflated 41%)
2023-01-11T22:03:20.3092793Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211747.xml (deflated 41%)
2023-01-11T22:03:20.3093379Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211751.xml (deflated 41%)
2023-01-11T22:03:20.3093986Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211755.xml (deflated 41%)
2023-01-11T22:03:20.3094581Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211759.xml (deflated 42%)
2023-01-11T22:03:20.3095162Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211803.xml (deflated 42%)
2023-01-11T22:03:20.3095742Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211807.xml (deflated 41%)
2023-01-11T22:03:20.3096320Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211811.xml (deflated 42%)
2023-01-11T22:03:20.3096904Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211815.xml (deflated 42%)
2023-01-11T22:03:20.3097488Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211819.xml (deflated 41%)
2023-01-11T22:03:20.3098073Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211820.xml (deflated 41%)
2023-01-11T22:03:20.3098521Z   adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%)
2023-01-11T22:03:20.3124347Z ##[group]Run # Remove any previous test reports if they exist
2023-01-11T22:03:20.3124640Z # Remove any previous test reports if they exist
2023-01-11T22:03:20.3124862Z rm -f usage-log-*.zip
2023-01-11T22:03:20.3125140Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2023-01-11T22:03:20.3125428Z # so check to see if the file exists first
2023-01-11T22:03:20.3125642Z if [ -f 'usage_log.txt' ]; then
2023-01-11T22:03:20.3125887Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2023-01-11T22:03:20.3126102Z fi
2023-01-11T22:03:20.3137129Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:03:20.3137352Z env:
2023-01-11T22:03:20.3137528Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:03:20.3137803Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:03:20.3138115Z   FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_10588877413
2023-01-11T22:03:20.3138351Z ##[endgroup]
2023-01-11T22:03:20.3703214Z   adding: usage_log.txt (deflated 95%)
2023-01-11T22:03:20.3760935Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T22:03:20.3761153Z with:
2023-01-11T22:03:20.3761357Z   s3-prefix: pytorch/pytorch/3896099317/2/artifact

2023-01-11T22:03:20.3761564Z   retention-days: 14
2023-01-11T22:03:20.3761754Z   if-no-files-found: warn
2023-01-11T22:03:20.3761953Z   path: test-jsons-*.zip
2023-01-11T22:03:20.3762123Z   name: artifact
2023-01-11T22:03:20.3762302Z   s3-bucket: gha-artifacts
2023-01-11T22:03:20.3762491Z   region: us-east-1
2023-01-11T22:03:20.3762651Z env:
2023-01-11T22:03:20.3762911Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:03:20.3763179Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:03:20.3763441Z ##[endgroup]
2023-01-11T22:03:20.7465184Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T22:03:20.7465479Z With the provided path, there will be 1 file uploaded
2023-01-11T22:03:20.7465747Z Uploading to s3 prefix: pytorch/pytorch/3896099317/2/artifact
2023-01-11T22:03:20.7472609Z Starting upload of test-jsons-test-distributed-1-2-linux.2xlarge_10588877413.zip
2023-01-11T22:03:20.8452785Z Finished upload of test-jsons-test-distributed-1-2-linux.2xlarge_10588877413.zip
2023-01-11T22:03:20.8596179Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T22:03:20.8596394Z with:
2023-01-11T22:03:20.8596592Z   s3-prefix: pytorch/pytorch/3896099317/2/artifact

2023-01-11T22:03:20.8596806Z   retention-days: 14
2023-01-11T22:03:20.8596986Z   if-no-files-found: error
2023-01-11T22:03:20.8597530Z   path: test-reports-*.zip
2023-01-11T22:03:20.8597730Z   name: artifact
2023-01-11T22:03:20.8597894Z   s3-bucket: gha-artifacts
2023-01-11T22:03:20.8598078Z   region: us-east-1
2023-01-11T22:03:20.8598239Z env:
2023-01-11T22:03:20.8598396Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:03:20.8598664Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:03:20.8598993Z ##[endgroup]
2023-01-11T22:03:21.2021998Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T22:03:21.2022394Z With the provided path, there will be 1 file uploaded
2023-01-11T22:03:21.2022673Z Uploading to s3 prefix: pytorch/pytorch/3896099317/2/artifact
2023-01-11T22:03:21.2029986Z Starting upload of test-reports-test-distributed-1-2-linux.2xlarge_10588877413.zip
2023-01-11T22:03:21.3718764Z Finished upload of test-reports-test-distributed-1-2-linux.2xlarge_10588877413.zip
2023-01-11T22:03:21.3846346Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T22:03:21.3846566Z with:
2023-01-11T22:03:21.3846779Z   s3-prefix: pytorch/pytorch/3896099317/2/artifact

2023-01-11T22:03:21.3846996Z   retention-days: 14
2023-01-11T22:03:21.3847194Z   if-no-files-found: ignore
2023-01-11T22:03:21.3847395Z   path: usage-log-*.zip
2023-01-11T22:03:21.3847562Z   name: artifact
2023-01-11T22:03:21.3847746Z   s3-bucket: gha-artifacts
2023-01-11T22:03:21.3847937Z   region: us-east-1
2023-01-11T22:03:21.3848095Z env:
2023-01-11T22:03:21.3848268Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:03:21.3848541Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:03:21.3848784Z ##[endgroup]
2023-01-11T22:03:21.7318586Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T22:03:21.7319138Z With the provided path, there will be 1 file uploaded
2023-01-11T22:03:21.7319602Z Uploading to s3 prefix: pytorch/pytorch/3896099317/2/artifact
2023-01-11T22:03:21.7327975Z Starting upload of usage-log-test-distributed-1-2-linux.2xlarge_10588877413.zip
2023-01-11T22:03:21.8919278Z Finished upload of usage-log-test-distributed-1-2-linux.2xlarge_10588877413.zip
2023-01-11T22:03:21.9043984Z ##[group]Run # shellcheck disable=SC2156
2023-01-11T22:03:21.9044233Z # shellcheck disable=SC2156
2023-01-11T22:03:21.9044527Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2023-01-11T22:03:21.9055830Z shell: /usr/bin/bash -e {0}
2023-01-11T22:03:21.9056012Z env:
2023-01-11T22:03:21.9056174Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:03:21.9056448Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:03:21.9056702Z ##[endgroup]
2023-01-11T22:03:23.8920631Z ##[group]Run set -x
2023-01-11T22:03:23.8920843Z set -x
2023-01-11T22:03:23.8921050Z python3 -m pip install -r requirements.txt
2023-01-11T22:03:23.8921301Z python3 -m pip install boto3==1.19.12
2023-01-11T22:03:23.8921594Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2023-01-11T22:03:23.8932840Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:03:23.8933050Z env:
2023-01-11T22:03:23.8933230Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:03:23.8933509Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:03:23.8933769Z   AWS_DEFAULT_REGION: us-east-1
2023-01-11T22:03:23.8933983Z   BRANCH: pull/91627
2023-01-11T22:03:23.8934247Z   TEST_CONFIG: distributed
2023-01-11T22:03:23.8934420Z   SHARD_NUMBER: 1
2023-01-11T22:03:23.8934633Z   BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7
2023-01-11T22:03:23.8934847Z   PR_NUMBER: 91627
2023-01-11T22:03:23.8935057Z   PYTORCH_RETRY_TEST_CASES: 1
2023-01-11T22:03:23.8935266Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2023-01-11T22:03:23.8935489Z   SHA1: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T22:03:23.8935692Z   TAG: 
2023-01-11T22:03:23.8935857Z   WORKFLOW_ID: 3896099317
2023-01-11T22:03:23.8936174Z   GITHUB_TOKEN: ***
2023-01-11T22:03:23.8936365Z   GHA_WORKFLOW_JOB_ID: 10588877413
2023-01-11T22:03:23.8936562Z ##[endgroup]
2023-01-11T22:03:23.8961834Z + python3 -m pip install -r requirements.txt
2023-01-11T22:03:24.1083796Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T22:03:24.1369221Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3)
2023-01-11T22:03:24.1397771Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.4)
2023-01-11T22:03:24.1406990Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2)
2023-01-11T22:03:24.1416030Z Requirement already satisfied: hypothesis in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (6.62.0)
2023-01-11T22:03:24.1778554Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.21.6)
2023-01-11T22:03:24.1789340Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2023-01-11T22:03:24.1865143Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.0)
2023-01-11T22:03:24.1874076Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2023-01-11T22:03:24.2126382Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2023-01-11T22:03:24.2293197Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (1.16.0)
2023-01-11T22:03:24.2303273Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (0.6.6)
2023-01-11T22:03:24.2309846Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (4.4.0)
2023-01-11T22:03:24.2320506Z Requirement already satisfied: sympy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.10.1)
2023-01-11T22:03:24.2339436Z Requirement already satisfied: filelock in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 15)) (3.9.0)
2023-01-11T22:03:24.2411515Z Requirement already satisfied: networkx in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 16)) (2.6.3)
2023-01-11T22:03:24.2568591Z Requirement already satisfied: jinja2 in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 17)) (3.1.2)
2023-01-11T22:03:24.2595353Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.38.4)
2023-01-11T22:03:24.2611923Z Requirement already satisfied: attrs>=19.2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (22.2.0)
2023-01-11T22:03:24.2871081Z Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (2.4.0)
2023-01-11T22:03:24.2883515Z Requirement already satisfied: exceptiongroup>=1.0.0; python_version < "3.11" in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (1.1.0)
2023-01-11T22:03:24.2902460Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2023-01-11T22:03:24.2924774Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.12.7)
2023-01-11T22:03:24.2934576Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.13)
2023-01-11T22:03:24.3087562Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2023-01-11T22:03:24.3100059Z Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/.local/lib/python3.7/site-packages (from sympy->-r requirements.txt (line 14)) (1.2.1)
2023-01-11T22:03:24.3156869Z Requirement already satisfied: MarkupSafe>=2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from jinja2->-r requirements.txt (line 17)) (2.1.1)
2023-01-11T22:03:24.3882288Z + python3 -m pip install boto3==1.19.12
2023-01-11T22:03:24.5963349Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T22:03:24.6144252Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12)
2023-01-11T22:03:24.6192949Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0)
2023-01-11T22:03:24.6205286Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12)
2023-01-11T22:03:24.6247032Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2)
2023-01-11T22:03:24.6281305Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2023-01-11T22:03:24.6301022Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.13)
2023-01-11T22:03:24.6452553Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2023-01-11T22:03:24.8315283Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2023-01-11T22:05:35.9057311Z [scribe] Scribe access token not provided, sending report via boto3...
2023-01-11T22:05:35.9058488Z ERROR ENCOUNTERED WHEN UPLOADING TO SCRIBE: {"errorMessage":"2023-01-11T22:05:26.766Z 7f2e3f7f-e399-4477-9569-f624d7089021 Task timed out after 60.02 seconds"}
2023-01-11T22:05:35.9058946Z 
2023-01-11T22:05:35.9061277Z ----- Historic stats comparison result ------
2023-01-11T22:05:35.9062031Z 
2023-01-11T22:05:35.9062217Z     job: linux-focal-py3.7-gcc7
2023-01-11T22:05:35.9062485Z     commit: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T22:05:35.9062641Z 
2023-01-11T22:05:35.9064099Z Commit graph (base is most recent master ancestor with at least one S3 report):
2023-01-11T22:05:35.9064422Z 
2023-01-11T22:05:35.9064521Z     : (master)
2023-01-11T22:05:35.9065039Z     |
2023-01-11T22:05:35.9065408Z     | * 8419ddda87 (HEAD)            total time  2148.96s
2023-01-11T22:05:35.9065759Z     | |
2023-01-11T22:05:35.9065954Z     | : (2 commits)
2023-01-11T22:05:35.9066117Z     |/
2023-01-11T22:05:35.9068223Z     * db2a237763 (base)   6 reports, total time  4537.52s ± 3598.58s
2023-01-11T22:05:35.9068655Z     * 2b0abd4ce3          6 reports, total time  4508.29s ± 3592.59s
2023-01-11T22:05:35.9068972Z     * f7939b21e1         18 reports, total time  2831.48s ± 3247.60s
2023-01-11T22:05:35.9069288Z     * cb3204823e          6 reports, total time  3677.58s ± 3486.86s
2023-01-11T22:05:35.9069618Z     * 6e236553f5          6 reports, total time  3702.23s ± 3507.16s
2023-01-11T22:05:35.9069930Z     * cce577b391          6 reports, total time  3768.82s ± 3640.72s
2023-01-11T22:05:35.9070241Z     * fae821c2f1          6 reports, total time  4476.74s ± 3618.76s
2023-01-11T22:05:35.9070546Z     * 0c3659586d          6 reports, total time  4420.82s ± 3571.24s
2023-01-11T22:05:35.9070858Z     * 122245985a          6 reports, total time  4479.72s ± 3631.94s
2023-01-11T22:05:35.9071160Z     * b797a24259          6 reports, total time  4511.06s ± 3679.54s
2023-01-11T22:05:35.9071339Z     |
2023-01-11T22:05:35.9071488Z     :
2023-01-11T22:05:35.9071580Z 
2023-01-11T22:05:35.9071700Z Removed  (across 1387 suites)     0 tests, totaling     0.00s
2023-01-11T22:05:35.9071958Z Modified (across    0 suites)     0 tests, totaling     0.00s
2023-01-11T22:05:35.9072203Z Added    (across   92 suites)  1190 tests, totaling +2559.91s
2023-01-11T22:05:35.9745852Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2023-01-11T22:05:35.9746079Z with:
2023-01-11T22:05:35.9746235Z env:
2023-01-11T22:05:35.9746406Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:05:35.9746681Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:05:35.9746922Z ##[endgroup]
2023-01-11T22:05:35.9762012Z ##[group]Run set -eou pipefail
2023-01-11T22:05:35.9762231Z set -eou pipefail
2023-01-11T22:05:35.9762411Z 
2023-01-11T22:05:35.9762645Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2023-01-11T22:05:35.9762882Z for _ in $(seq 1440); do
2023-01-11T22:05:35.9763102Z     # Break if no ssh session exists anymore
2023-01-11T22:05:35.9763320Z     if [ "$(who)" = "" ]; then
2023-01-11T22:05:35.9763500Z       break
2023-01-11T22:05:35.9763652Z     fi
2023-01-11T22:05:35.9763851Z     echo "."
2023-01-11T22:05:35.9764010Z     sleep 5
2023-01-11T22:05:35.9764175Z done
2023-01-11T22:05:35.9775376Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:05:35.9775583Z env:
2023-01-11T22:05:35.9775759Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:05:35.9776035Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:05:35.9776287Z ##[endgroup]
2023-01-11T22:05:35.9800863Z Holding runner for 2 hours until all ssh sessions have logged out
2023-01-11T22:05:35.9902459Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2023-01-11T22:05:35.9902761Z # ignore expansion of "docker ps -q" since it could be empty
2023-01-11T22:05:35.9903014Z # shellcheck disable=SC2046
2023-01-11T22:05:35.9903241Z docker stop $(docker ps -q) || true
2023-01-11T22:05:35.9903473Z # Prune all of the docker images
2023-01-11T22:05:35.9903676Z docker system prune -af
2023-01-11T22:05:35.9914215Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:05:35.9914436Z env:
2023-01-11T22:05:35.9914613Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:05:35.9914878Z   DOCKER_CONTAINER_ID: ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:05:35.9915138Z ##[endgroup]
2023-01-11T22:05:36.3800001Z ecc7409477c1
2023-01-11T22:05:37.6131883Z Deleted Containers:
2023-01-11T22:05:37.6132424Z ecc7409477c1c13bd051697665e355e23f6e9e1b37f4df9c9326e62bf1114747
2023-01-11T22:05:37.6132804Z 
2023-01-11T22:05:45.0686179Z Deleted Images:
2023-01-11T22:05:45.0687072Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T22:05:45.0687786Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7@sha256:d201d5d219c1681863b90b8d50e8b2382dd0a533dc69c772527c8255937144a4
2023-01-11T22:05:45.0688533Z deleted: sha256:872545647a28b88516eae4000db04fb3df455244b9807860ebbad7d21df7fbd3
2023-01-11T22:05:45.0689010Z deleted: sha256:0913ba780396d26b1a734b3dee7f02947640c13b56c904623d8d169cb2f445d9
2023-01-11T22:05:45.0689332Z deleted: sha256:2ab5e0d230803b5fd25b7b0f930eee8ab6c77cfac036eaa85c95a435b426df7e
2023-01-11T22:05:45.0689834Z deleted: sha256:058992102b05c2e3259d499eee2fb79686b842ff3b38268b218b9c9049bb4f4e
2023-01-11T22:05:45.0690316Z deleted: sha256:a3881c186a0b72148266b6499bd71e5a317b24fee69b6437d44f3c4975960695
2023-01-11T22:05:45.0690886Z deleted: sha256:13ab35328da38a4819bf04cd02facc248dfbaa82d5ec3a3b6e2c669dd985461b
2023-01-11T22:05:45.0691427Z deleted: sha256:aa2b1c0d1223bd8e06c4b3dd1f62593d69cd9d16cf4c46502020dd0ddabaedf4
2023-01-11T22:05:45.0692044Z deleted: sha256:07d56fb6f87c59cf211fb1d48faf0c0697dba1c8bd2eb68cfc32806d0660fb0b
2023-01-11T22:05:45.0692659Z deleted: sha256:b24802dbddd392269b952ab58802f68ad557e19cbce32bb194ab1c09e7eca6c5
2023-01-11T22:05:45.0693265Z deleted: sha256:ef000ff41666363f8fdeac09f9203e47a0ebaed82d61cd2aef4e648d75859d7c
2023-01-11T22:05:45.0693601Z deleted: sha256:b654f485842eed85c133b3d43fef43bfda9f7e718584e49590fa1ab94cbdb621
2023-01-11T22:05:45.0693923Z deleted: sha256:db407d4b58006712031749736db1c37874e5d4e908dfcba929dea8da63234d10
2023-01-11T22:05:45.0694240Z deleted: sha256:d93740e501b928fd7be659a8e1586a3d87e569c1544c2761ecdea7ab0997e7cc
2023-01-11T22:05:45.0694555Z deleted: sha256:ac19b11a34be7cb4598050f9aabd16ff0ff79831f485fb6000af0dfaddb436de
2023-01-11T22:05:45.0694884Z deleted: sha256:a5ae6c04d7731515ac814d89d2986f7910d0867922320cb22d751b3417011f43
2023-01-11T22:05:45.0695217Z deleted: sha256:6eb6bd31a4bd4d9a467b64ebdd39d775cba0ea1edb1de0b00ed3837d816425a8
2023-01-11T22:05:45.0695523Z deleted: sha256:19787711222d55795f879841531b67c7d4278c9f68c7c9510f023efc7e761faa
2023-01-11T22:05:45.0695880Z deleted: sha256:8fa2cc4e3117a1ec0a6d95bd9f17b254dbb0def34fcb15b3e7d6bb9a9f3e35c5
2023-01-11T22:05:45.0696216Z deleted: sha256:098023de3c889616f6862e6be3ec1c96adb5067276efcd2567245b9e39ea0fd9
2023-01-11T22:05:45.0696549Z deleted: sha256:2812d79bbc620bed5ab85a40ba81afc41f5c8b47dead2c7ae008f0dbf6a08d64
2023-01-11T22:05:45.0696866Z deleted: sha256:724cd734f88329c39fb2001c8b92ec22e9d9047f041df8d32978044c71884f31
2023-01-11T22:05:45.0697328Z deleted: sha256:8afdf6dfb1a843616a6386809d2ac8f394c29991357a9fde65af4382ebbe8ef2
2023-01-11T22:05:45.0697650Z deleted: sha256:ac10867395ba90a5662da28be64822423bde7ca92a50e646804c516a98e53004
2023-01-11T22:05:45.0697966Z deleted: sha256:d923c532b32b0d27a1fca08eb52224541269676efc35421ca1143f3bb64c0e7a
2023-01-11T22:05:45.0698276Z deleted: sha256:166fd4df8a671ab6d72685e787eb87bb17716ecfb8383732ce0330cc01de9366
2023-01-11T22:05:45.0698597Z deleted: sha256:0cc9007352917a0fe743d12ca9d2333b9123ddd5d501f9caee7281d837d5b287
2023-01-11T22:05:45.0698936Z deleted: sha256:ad1d657714a0c9feee756d8598c9ebdd7613bea5d3c53cf0a8ec89eb2c27ba2a
2023-01-11T22:05:45.0699261Z deleted: sha256:d4345474a810d8add867c09fddec1703c6f1ef96056681cbd404c3adf9c26231
2023-01-11T22:05:45.0699591Z deleted: sha256:90cfc4251be8e59835753e6518aefddf4f09951dde250ea0f3e10a09cd1bbef1
2023-01-11T22:05:45.0699920Z deleted: sha256:0f39cccb089ed0a8c80302e51e04579017d34266b2dbbf7bc09a826071de9da5
2023-01-11T22:05:45.0700228Z deleted: sha256:844c9188ba4b07d8d846104595214e63ec0c9818848540249393ced49196e702
2023-01-11T22:05:45.0700521Z deleted: sha256:b60329deafe4635066b5ce4cc72f1a3d11902a643666ffa7366b92b1e4c55f80
2023-01-11T22:05:45.0700849Z deleted: sha256:3545474c90ffd46a1ca93bbf91fa70bbbf7cdf7e0085a5e4e158164104c753ff
2023-01-11T22:05:45.0701265Z deleted: sha256:df43d5787137e8e2383f7d1627a5093edba3534745c272878d97083fdbf5f984
2023-01-11T22:05:45.0701582Z deleted: sha256:d9b003a92bbdfabe507ef0f8c00445a56c47fcf6f50e58ada5e86a6992f49c09
2023-01-11T22:05:45.0702118Z deleted: sha256:05d44c5f8014dfc2a6fb85fcaf86916e1305cda1571ee0f1e6320de59ce05bd7
2023-01-11T22:05:45.0702447Z deleted: sha256:530d276655c95d43dd64bb980673a2b13ebdb698a82a066a9b9fdf2e59dfe0bb
2023-01-11T22:05:45.0702767Z deleted: sha256:0daee3895574fbd6e661c964d60581233c5a8589b84d2590423deb3b29b51a46
2023-01-11T22:05:45.0703077Z deleted: sha256:8c76fd3e05eca362fe3d75f5a4c7133bd502ee43f59f17c64ed0c0c22fd46016
2023-01-11T22:05:45.0703395Z deleted: sha256:acb983c614c2804059e0970d18e5464a7542b4d551448270b8fd78cd782e6aa4
2023-01-11T22:05:45.0703719Z deleted: sha256:fda9d1b52f2ba36ac6b2b465a90f8982d9be82c45e4157ae554cde955389fcb3
2023-01-11T22:05:45.0704032Z deleted: sha256:1801c74cef4832dd65d4127c037ae1051a2a1eb90e7ce8120146851ace5946c7
2023-01-11T22:05:45.0704356Z deleted: sha256:240d82bb34e0f78bd9f1a0fc9663c2ddeb81020886c57c7188bb5b70b99e22ee
2023-01-11T22:05:45.0704680Z deleted: sha256:f88a7257109761d6e36e1e2de77e614d3de016b149db7113bcd8662c1fe99370
2023-01-11T22:05:45.0704986Z deleted: sha256:8a214e9e410e6400b71df4b1490fb8f2873203bc232689be875400d89f80b4c2
2023-01-11T22:05:45.0705286Z deleted: sha256:99332f27e027fff91df110db38b2f98a3b9c9212d130aef9a9ace15b4ca675a4
2023-01-11T22:05:45.0705610Z deleted: sha256:5e466b1d33a7b6f612f0d14153f80a4b2aaafd042765ec629a7099ed5d6b54a0
2023-01-11T22:05:45.0706025Z deleted: sha256:91eefd95450265ebde00db0268894212aeccff9e26fa2337760b1d75d20e4173
2023-01-11T22:05:45.0706351Z deleted: sha256:69463c6d56ebcccb292eff675b4d3f48de45fb09ecbb7847f9ccbe876d7e3504
2023-01-11T22:05:45.0706686Z deleted: sha256:296f50ac773cc9a4c23ba55c2564c926765f3c7d0a3fe6fe47e29aaa9ec9b22e
2023-01-11T22:05:45.0707015Z deleted: sha256:f89150a51cf7591d0e1a9f53ed7ea0c3aff94a467e80176a2d4f55c0e3bb41da
2023-01-11T22:05:45.0707333Z deleted: sha256:6d34afa8d6b4573ee5ab2c08738d797c78ad53a584f42d0ad9686080cb1cf955
2023-01-11T22:05:45.0707660Z deleted: sha256:77f6edeb37e215f3a81aea6d82d9c232700f550fb52211911ed569381c45de60
2023-01-11T22:05:45.0707973Z deleted: sha256:8744929d3f79f26f92321dfad103e20ab4d6649223c519635b8091c03e79bf23
2023-01-11T22:05:45.0708278Z deleted: sha256:cbcf68337e6007e98da7234219d686d9865e3691796e71ea82b931bcccbb3bcd
2023-01-11T22:05:45.0708578Z deleted: sha256:5ef8fcc78725941709d1712da344c0bc9f22450dde23c9da2c640dce6f58dc62
2023-01-11T22:05:45.0708906Z deleted: sha256:017f8ab4193b43fb8adcceda4b63e7a0e77f98591b0a0d8e2230cd43051d7651
2023-01-11T22:05:45.0709256Z deleted: sha256:b4c50db5a4dbcf2c4cf6b0e5c19878b2cd52e983fb5366ce83d7fd8e2539313a
2023-01-11T22:05:45.0709574Z deleted: sha256:0002c93bdb3704dd9e36ce5153ef637f84de253015f3ee330468dccdeacad60b
2023-01-11T22:05:45.0709757Z 
2023-01-11T22:05:45.0716184Z Total reclaimed space: 9.443GB
2023-01-11T22:05:45.0827268Z Post job cleanup.
2023-01-11T22:05:45.0899394Z Post job cleanup.
2023-01-11T22:05:45.1993896Z [command]/usr/bin/git version
2023-01-11T22:05:45.2040209Z git version 2.38.1
2023-01-11T22:05:45.2077897Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/0415bc31-9024-46da-9f9e-6583d8128487' before making global git config changes
2023-01-11T22:05:45.2078536Z Adding repository directory to the temporary git global config as a safe directory
2023-01-11T22:05:45.2082196Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T22:05:45.2127898Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-01-11T22:05:45.2157261Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2023-01-11T22:05:45.2593575Z Entering 'android/libs/fbjni'
2023-01-11T22:05:45.2627719Z Entering 'third_party/FP16'
2023-01-11T22:05:45.2661933Z Entering 'third_party/FXdiv'
2023-01-11T22:05:45.2695020Z Entering 'third_party/NNPACK'
2023-01-11T22:05:45.2729703Z Entering 'third_party/QNNPACK'
2023-01-11T22:05:45.2763904Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T22:05:45.2797337Z Entering 'third_party/XNNPACK'
2023-01-11T22:05:45.2843775Z Entering 'third_party/benchmark'
2023-01-11T22:05:45.2876904Z Entering 'third_party/cpuinfo'
2023-01-11T22:05:45.2911549Z Entering 'third_party/cub'
2023-01-11T22:05:45.2944261Z Entering 'third_party/cudnn_frontend'
2023-01-11T22:05:45.2982705Z Entering 'third_party/cutlass'
2023-01-11T22:05:45.3023156Z Entering 'third_party/eigen'
2023-01-11T22:05:45.3057518Z Entering 'third_party/fbgemm'
2023-01-11T22:05:45.3091079Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T22:05:45.3125234Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T22:05:45.3158240Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T22:05:45.3191388Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T22:05:45.3225967Z Entering 'third_party/flatbuffers'
2023-01-11T22:05:45.3262357Z Entering 'third_party/fmt'
2023-01-11T22:05:45.3296539Z Entering 'third_party/foxi'
2023-01-11T22:05:45.3331261Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T22:05:45.3365305Z Entering 'third_party/gloo'
2023-01-11T22:05:45.3398956Z Entering 'third_party/googletest'
2023-01-11T22:05:45.3435017Z Entering 'third_party/ideep'
2023-01-11T22:05:45.3468958Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T22:05:45.3504742Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T22:05:45.3546329Z Entering 'third_party/ios-cmake'
2023-01-11T22:05:45.3581077Z Entering 'third_party/ittapi'
2023-01-11T22:05:45.3616806Z Entering 'third_party/kineto'
2023-01-11T22:05:45.3650530Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T22:05:45.3685361Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T22:05:45.3721086Z Entering 'third_party/nccl/nccl'
2023-01-11T22:05:45.3757026Z Entering 'third_party/neon2sse'
2023-01-11T22:05:45.3791457Z Entering 'third_party/nlohmann'
2023-01-11T22:05:45.3828162Z Entering 'third_party/onnx'
2023-01-11T22:05:45.3874092Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T22:05:45.3909604Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T22:05:45.3945558Z Entering 'third_party/onnx-tensorrt'
2023-01-11T22:05:45.3980614Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T22:05:45.4019281Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T22:05:45.4054006Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T22:05:45.4087692Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T22:05:45.4126540Z Entering 'third_party/pocketfft'
2023-01-11T22:05:45.4160536Z Entering 'third_party/protobuf'
2023-01-11T22:05:45.4198774Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T22:05:45.4232402Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T22:05:45.4268726Z Entering 'third_party/psimd'
2023-01-11T22:05:45.4303580Z Entering 'third_party/pthreadpool'
2023-01-11T22:05:45.4337602Z Entering 'third_party/pybind11'
2023-01-11T22:05:45.4372340Z Entering 'third_party/python-enum'
2023-01-11T22:05:45.4407447Z Entering 'third_party/python-peachpy'
2023-01-11T22:05:45.4440984Z Entering 'third_party/python-six'
2023-01-11T22:05:45.4475313Z Entering 'third_party/sleef'
2023-01-11T22:05:45.4509292Z Entering 'third_party/tbb'
2023-01-11T22:05:45.4546368Z Entering 'third_party/tensorpipe'
2023-01-11T22:05:45.4583329Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T22:05:45.4616780Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T22:05:45.4650077Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T22:05:45.4684737Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T22:05:45.4718157Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T22:05:45.4755235Z Entering 'third_party/zstd'
2023-01-11T22:05:45.4803531Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-01-11T22:05:45.4830316Z http.https://github.com/.extraheader
2023-01-11T22:05:45.4837997Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2023-01-11T22:05:45.4870179Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2023-01-11T22:05:45.5128531Z Entering 'android/libs/fbjni'
2023-01-11T22:05:45.5149553Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5177225Z Entering 'third_party/FP16'
2023-01-11T22:05:45.5197963Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5224346Z Entering 'third_party/FXdiv'
2023-01-11T22:05:45.5244506Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5279926Z Entering 'third_party/NNPACK'
2023-01-11T22:05:45.5300433Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5328578Z Entering 'third_party/QNNPACK'
2023-01-11T22:05:45.5348975Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5375707Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T22:05:45.5395403Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5423278Z Entering 'third_party/XNNPACK'
2023-01-11T22:05:45.5442802Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5479987Z Entering 'third_party/benchmark'
2023-01-11T22:05:45.5500006Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5526816Z Entering 'third_party/cpuinfo'
2023-01-11T22:05:45.5547184Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5574687Z Entering 'third_party/cub'
2023-01-11T22:05:45.5595026Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5635313Z Entering 'third_party/cudnn_frontend'
2023-01-11T22:05:45.5656115Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5687836Z Entering 'third_party/cutlass'
2023-01-11T22:05:45.5708492Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5742296Z Entering 'third_party/eigen'
2023-01-11T22:05:45.5762614Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5791495Z Entering 'third_party/fbgemm'
2023-01-11T22:05:45.5811555Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5840131Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T22:05:45.5859999Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5887336Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T22:05:45.5907753Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5935088Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T22:05:45.5954745Z http.https://github.com/.extraheader
2023-01-11T22:05:45.5981297Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T22:05:45.6002624Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6029787Z Entering 'third_party/flatbuffers'
2023-01-11T22:05:45.6049651Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6077938Z Entering 'third_party/fmt'
2023-01-11T22:05:45.6098500Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6124021Z Entering 'third_party/foxi'
2023-01-11T22:05:45.6144448Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6170220Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T22:05:45.6191137Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6217994Z Entering 'third_party/gloo'
2023-01-11T22:05:45.6238415Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6265356Z Entering 'third_party/googletest'
2023-01-11T22:05:45.6284798Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6324110Z Entering 'third_party/ideep'
2023-01-11T22:05:45.6346735Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6373060Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T22:05:45.6392812Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6421928Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T22:05:45.6441463Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6483577Z Entering 'third_party/ios-cmake'
2023-01-11T22:05:45.6505304Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6544449Z Entering 'third_party/ittapi'
2023-01-11T22:05:45.6565152Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6591087Z Entering 'third_party/kineto'
2023-01-11T22:05:45.6611492Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6638144Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T22:05:45.6658304Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6684639Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T22:05:45.6704436Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6732290Z Entering 'third_party/nccl/nccl'
2023-01-11T22:05:45.6752907Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6778863Z Entering 'third_party/neon2sse'
2023-01-11T22:05:45.6798564Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6825134Z Entering 'third_party/nlohmann'
2023-01-11T22:05:45.6845414Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6873275Z Entering 'third_party/onnx'
2023-01-11T22:05:45.6893713Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6932380Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T22:05:45.6953450Z http.https://github.com/.extraheader
2023-01-11T22:05:45.6980418Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T22:05:45.7000647Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7042392Z Entering 'third_party/onnx-tensorrt'
2023-01-11T22:05:45.7063185Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7088723Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T22:05:45.7108871Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7148222Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T22:05:45.7168015Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7194469Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T22:05:45.7213590Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7240802Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T22:05:45.7260090Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7291234Z Entering 'third_party/pocketfft'
2023-01-11T22:05:45.7311123Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7337618Z Entering 'third_party/protobuf'
2023-01-11T22:05:45.7357715Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7388001Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T22:05:45.7407370Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7434059Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T22:05:45.7453731Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7481998Z Entering 'third_party/psimd'
2023-01-11T22:05:45.7503215Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7529536Z Entering 'third_party/pthreadpool'
2023-01-11T22:05:45.7550372Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7576687Z Entering 'third_party/pybind11'
2023-01-11T22:05:45.7596353Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7623519Z Entering 'third_party/python-enum'
2023-01-11T22:05:45.7642507Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7669391Z Entering 'third_party/python-peachpy'
2023-01-11T22:05:45.7689090Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7724248Z Entering 'third_party/python-six'
2023-01-11T22:05:45.7745032Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7771229Z Entering 'third_party/sleef'
2023-01-11T22:05:45.7791571Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7818430Z Entering 'third_party/tbb'
2023-01-11T22:05:45.7839248Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7867249Z Entering 'third_party/tensorpipe'
2023-01-11T22:05:45.7887471Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7914847Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T22:05:45.7933563Z http.https://github.com/.extraheader
2023-01-11T22:05:45.7960256Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T22:05:45.7979743Z http.https://github.com/.extraheader
2023-01-11T22:05:45.8006118Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T22:05:45.8026234Z http.https://github.com/.extraheader
2023-01-11T22:05:45.8051952Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T22:05:45.8072288Z http.https://github.com/.extraheader
2023-01-11T22:05:45.8098709Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T22:05:45.8118293Z http.https://github.com/.extraheader
2023-01-11T22:05:45.8147842Z Entering 'third_party/zstd'
2023-01-11T22:05:45.8168187Z http.https://github.com/.extraheader
2023-01-11T22:05:45.8396890Z Cleaning up orphan processes